SAM

Overview

SAM, the Segment Anything Model, is a promptable segmentation foundation model released by Meta in 2023. You give it an image and a hint, such as a click on an object, a rough box or a rough mask, and it returns a mask for the thing you pointed at. It does this without being trained on your specific objects, which is why it is called zero-shot. Its masks are good enough that SAM became a standard tool for fast annotation, background removal and as a building block in larger vision systems.

SAM does not say what an object is. It finds where an object is and draws its outline. For names and classes you pair it with a classifier or an embedding model such as CLIP, or with a detector such as YOLO or DETR that supplies boxes. For closed-set instance segmentation trained on your own labels, Mask R-CNN is a classic choice. This page covers the idea, the history, the mechanics, a worked example and how segmentation output links to models that learn from raw tables.

What it is

SAM is a model, a task definition and a very large dataset released together. The task is promptable segmentation: given any prompt that says what to segment, return a valid mask.

The three parts of the project

• The task: promptable segmentation, where the model must return a sensible mask for any prompt, even an ambiguous one.
• The model: an image encoder, a prompt encoder and a lightweight mask decoder.
• The data: a large mask dataset called SA-1B, built with a model-in-the-loop data engine. It contains more than a billion masks over millions of images, according to the release.

Prompts it understands

• Points: one or more clicks, marked as foreground or background.
• Boxes: a rectangle around the target.
• Masks: a rough mask that the model refines.
• Text: the original release explored text prompts as an early experiment, and text-driven use today usually comes from pairing SAM with other models.

What it returns

For an ambiguous prompt, such as one click on a shirt, SAM can return several masks: the button, the shirt, the whole person. Each comes with a quality score. This ambiguity-aware design is intentional.

Everyday analogy

Think of an expert tracing assistant. You tap on a thing in a photo and say "this one". The assistant cuts around it with scissors. If you tapped on a sleeve, they may offer three cuts: the cuff, the sleeve, the whole jacket. They do not know the word "jacket". They just know where things begin and end.

Where it came from

Background

Segmentation research had long been split into semantic, instance and panoptic tasks, each with its own models and datasets. Interactive segmentation, where a user clicks and the model responds, existed but models were usually narrow. At the same time, language models had shown that pre-training a large model on broad data and then prompting it works across many tasks.

The foundation model idea for vision

The SAM paper, "Segment Anything" (2023), proposed bringing that prompting recipe to segmentation. The goals were a task flexible enough for pre-training, a model fast enough to be interactive, and a dataset large enough to teach general object boundaries.

The data engine

Masks were collected in three stages:

• Assisted manual: annotators click, the model helps, humans correct.
• Semi-automatic: the model finds confident masks automatically and annotators fill in missing objects.
• Fully automatic: the model prompts itself with a grid of points and produces masks for the whole image, followed by filtering for quality.

The model and the data improved each other in a loop. By the final stage, most masks were automatically generated.

Follow-ups

Later work extended the idea to video, to lighter models for mobile devices, to higher-quality masks on fine structures, and to systems that pair segmentation with text-based detectors. Names, versions and numbers evolve, so check current documentation for what is available and its license.

Impact

SAM changed how many teams label data. Instead of drawing polygons by hand, annotators click. It also became a default component in pipelines that need object cut-outs without training a new model.

How it works

SAM is designed so that the heavy computation happens once per image, and each prompt is then processed in a few milliseconds.

Step 1: Image encoder

A large vision transformer, pre-trained with a masked autoencoder method, converts the image into an embedding: a grid of feature vectors that describe the image content. This step is heavy and is done once per image. Analogy: reading and memorizing a map before anyone asks directions.

Step 2: Prompt encoder

Prompts are turned into vectors:

• Points and box corners become positional encodings plus a learned embedding for their type (foreground point, background point, box corner).
• Masks are processed by a small convolutional network and added to the image embedding.
• Text prompts, when used, come from a separate text encoder.

Analogy: your finger on the map is translated into coordinates the system understands.

Step 3: Mask decoder

A small transformer decoder takes the image embedding and the prompt embeddings and produces masks. It uses two-way attention: prompt tokens attend to the image embedding, and the image embedding attends back to the prompt tokens. This lets the prompt and the image refine each other. A few output tokens produce multiple mask candidates and a predicted quality score for each.

Step 4: Upsampling and output

The decoder output is upsampled to a mask at higher resolution. A threshold turns it into a binary mask. Multiple masks and scores are returned for ambiguous prompts.

Step 5: Ambiguity handling in training

During training the model predicts several masks per prompt, and the loss is computed against the ground truth using only the best-matching prediction. This lets the model keep different valid answers instead of averaging them into a blur. It is like asking three people to guess and rewarding the closest one, so they learn to cover different possibilities.

Step 6: Training loss

Masks are trained with a mix of focal loss and dice loss, which handle class imbalance between object pixels and background pixels. A separate head predicts the IoU of each mask, trained to match the real IoU, so the score can be used to pick or filter masks.

Step 7: Interactive training

Training simulates user interaction. The model gets one prompt, predicts, then receives extra clicks placed in the error regions, and predicts again. This teaches it to improve with each correction.

Reading the prompt types in more detail

A single foreground point is the cheapest prompt but the most ambiguous. Two or three points on the same object make the answer clearer. A background point tells the model "not this region", which helps when two objects touch. A box is more informative than a point because it gives extent as well as location. A rough mask is the most informative and is mostly used to refine a result.

• One point: fast, ambiguous.
• Several points: clearer, still quick.
• Box: usually the best single prompt.
• Box plus a background point: good for crowded areas.
• Previous mask as input: good for polishing.

What a good mask looks like

A good mask follows the visible boundary, has no holes where the object is solid, and contains no stray islands. Quality checks can be automated. Count connected components. Compute the ratio of mask area to the convex hull area. Check that the mask touches the prompt box on most sides if the box was tight. These checks are cheap and catch many failures.

Mask formats

• Binary array: simple but large.
• Run-length encoding: compact and common in datasets.
• Polygon: easy to edit by hand and to import into labeling tools.
• Signed distance or soft mask: useful if you plan to blend edges.

Converting between formats can lose detail. Keep the highest quality format as the source of truth.

Why the data engine mattered

A model-in-the-loop labeling process changes the economics of data. Annotators spend their time on hard cases because easy ones are handled by the model. As the model improves, the share of human effort falls. This idea applies beyond segmentation. Any domain with scarce labels can use a first rough model to propose labels that humans correct.

Automatic mask generation

To segment everything in an image without prompts, a grid of points is used as prompts. Each point yields masks. Low-quality masks are filtered by predicted IoU and stability checks, and duplicates are removed with non-maximum suppression. The result is a set of masks that cover most objects and parts. The set has no class names.

Diagnosing poor masks

• Mask leaks into the background: add a background point or tighten the box.
• Mask covers only part of the object: add foreground points on the missing part, or loosen the box.
• Mask includes a neighbor: add a background point on the neighbor.
• Mask is noisy at edges: raise the resolution of the input or apply a refinement step.
• Mask differs between runs: check for nondeterminism and fix thresholds.

Most issues are solved by a better prompt rather than a different model.

Scale of the problem

A typical image can have dozens of objects. Prompting each with a box from a detector takes one encoder pass and dozens of cheap decoder passes. That is far cheaper than dozens of independent model runs, and it is the reason the split design scales.

Why the split design is smart

The image encoder is heavy but runs once. The prompt encoder and mask decoder are small. In a labeling tool, the user waits a moment when opening an image, and then every click feels instant. The same split makes it practical to run many prompts on the same image, for example one per detected box.

Using boxes from a detector

A common pipeline: a detector such as YOLO or DETR finds boxes with class names. Each box is given to SAM as a prompt. SAM returns an accurate mask for the box. The result is class-aware instance segmentation without mask labels for training. The class comes from the detector, the shape from SAM.

Using text with other models

SAM itself is not a text-based detector. Systems that let you type "the red car" typically pair an open-vocabulary detector or a model like CLIP with SAM: the first finds candidate regions that match the words, and SAM cuts them out.

Limits of resolution

The mask decoder output is relatively low resolution before upsampling. Very thin structures, fine hair, wires and tiny parts can be rough. Higher-quality variants and refinement tools exist for this.

Key concepts and terms

A worked example

Goal: build a labeled dataset of defects on circuit boards quickly, using SAM to produce masks from boxes, then train a small model on the results.

Step 1: Get boxes

Use a detector, a human clicking, or a quick manual box draw to mark defects. Suppose we have 2,000 images with boxes and class names such as "solder bridge" and "missing part".

Step 2: Load SAM and set the image

The code below follows the pattern of the official open source library. Model type and checkpoint names are placeholders.

import numpy as np
from segment_anything import sam_model_registry, SamPredictor

sam = sam_model_registry["vit_placeholder"](checkpoint="sam_checkpoint.pth")
sam.to("cuda")
predictor = SamPredictor(sam)

predictor.set_image(image_rgb)   # heavy step, once per image

Step 3: Prompt with a box

box = np.array([x1, y1, x2, y2])  # pixel coordinates
masks, scores, logits = predictor.predict(
    box=box,
    multimask_output=False,   # one mask for a clear box prompt
)
mask = masks[0]               # boolean array, H x W
print(scores[0])

Step 4: Prompt with points

points = np.array([[320, 240], [350, 260]])
labels = np.array([1, 0])      # 1 = foreground, 0 = background
masks, scores, _ = predictor.predict(
    point_coords=points,
    point_labels=labels,
    multimask_output=True,     # ask for several candidates
)
best = masks[int(np.argmax(scores))]

Step 5: Quality filtering

Not every mask is good. Apply simple rules:

• Keep masks whose predicted score is above a threshold, such as 0.85.
• Reject masks that are far larger than their box.
• Reject masks with many disconnected parts when the object should be one piece.
• Send the rest to a human for a quick check.

Step 6: A numeric walk-through

Suppose a box covers 100 by 80 pixels, which is 8,000 pixels. The returned mask covers 3,200 pixels, a 40 percent fill. For a solder bridge, which is a thin blob inside a larger box, that is plausible. If the mask covered 7,900 pixels, it would nearly equal the box, which may mean SAM captured the board background instead. A rule such as "fill between 5 and 90 percent" would flag that case for review.

If the pixel size is 0.05 millimeters, 3,200 pixels equal 3,200 times 0.05 times 0.05, which is 8 square millimeters.

Step 7: Export labels

Convert masks to polygons or run-length encoding in the annotation format your training tool expects. Keep the class from the original box.

Step 8: Train a task model

Train a closed-set model on the new labels. For fast boxes, use YOLO. For production masks with a fixed class list, use Mask R-CNN. SAM served as the labeling accelerator, not the final runtime model.

Step 9: Write masks into a table

For each defect store image id, class, score, area in pixels and millimeters, bounding box, centroid and fill ratio. Roll up to per-board columns like total defect area and count. These feed tabular models for yield prediction.

Step 10: Review the results

Sample 100 annotations, check the masks by eye, and measure the share needing correction. If it is high for one class, add more clicks or refine with extra prompts for that class.

Strengths and limitations

Strengths

• Strong zero-shot masks on a wide variety of images.
• Fast interactive use after the one-time image encoding.
• Flexible prompts: points, boxes and masks.
• Ambiguity-aware outputs with quality scores.
• Great for speeding up labeling and for pipelines needing cut-outs.
• Open source code and checkpoints were released, which spread its use.
• Works as a general component with detectors and embedding models.

Limitations

• No class names. It does not know what the object is.
• The image encoder is heavy and needs a GPU for comfortable speed.
• Fine structures and thin shapes can be rough.
• Domain gaps exist: medical scans, microscopy, satellite images and industrial textures can need adaptation or fine-tuning.
• Automatic mode can over-segment, splitting objects into parts, or miss low-contrast objects.
• Results depend on prompts. Poor boxes or misplaced points give poor masks.
• Consistency across frames is not guaranteed for image-only use. Video variants address this.
• The license of the code and checkpoints must be checked before commercial use.

Honest remarks on evaluation

Zero-shot success on public images does not guarantee success on your domain. Measure mask IoU against a small hand-labeled sample from your own data before relying on SAM in a pipeline.

How it compares with alternatives

Typical combinations

• Detector plus SAM: boxes with classes become class-aware masks.
• CLIP plus SAM: SAM proposes masks, CLIP scores each crop against text.
• SAM plus a small classifier: label masks quickly, then train a model.
• SAM plus tracking: propagate masks across video frames.

Choosing quickly

• Need masks fast without labels: SAM.
• Need real-time class-aware masks on a fixed class list: train a dedicated model, perhaps with SAM-made labels.
• Need open-vocabulary words: pair with a text-aware model.

Role in tabular foundation models

SAM is a vision model and does not read tables. The connection to table-native foundation models is practical and conceptual. The pages on research, architecture and benchmarks describe how a model that learns directly from tables is built and judged.

Prompting and in-context learning

SAM's central idea is that a pre-trained model takes a prompt and adapts without further training. Tabular foundation models like TabPFN and TabICL do the same: the "prompt" is a table of labeled example rows, and the model predicts labels for new rows in a single forward pass. In both cases, heavy learning happens in pre-training, and use-time adaptation is cheap.

Heavy encoder, light decoder

SAM encodes the image once and answers many prompts cheaply. A table model can encode a context table once and reuse it for many query rows. That is a practical trick for serving: cache the encoded context, then score many rows quickly.

Data engines and synthetic data

SAM's data engine used the model to create its own training data. Tabular foundation models are often pre-trained on generated tables that mimic many real structures, see synthetic data generation. Both show a pattern: when real labeled data is scarce, build a loop or a generator that produces diverse training worlds.

Masked pre-training

The SAM image encoder was initialized with masked autoencoding: hide patches and learn to rebuild them. Tabular analogs hide cells and learn to restore them, as in masked cell modeling and denoising autoencoders. This is also the basis for imputation of missing cells, see imputation networks.

Segmentation output becomes columns

Each mask can yield numbers: area, perimeter, elongation, mean color, texture statistics, distance to neighbors. Per image, the number of masks varies, so teams aggregate to fixed columns. The result is a mixed-type table with numeric, categorical and missing cells. Table-native models handle this directly, even if column names are unclear.

Missing values

When a mask is rejected by quality filters, its measures are missing. Treating that as missing, not as zero, keeps meaning intact. A table model with an imputation head or native missing-value handling can fill or flag these cells.

Anomaly detection

Distributions of mask sizes, counts and shapes shift when a process drifts. Use isolation forest, one-class SVM or the general anomaly detection page for ideas. A tabular foundation model can also score rows by how unlikely they are given normal context rows.

Semantic and sector classification

SAM gives shapes without names. Names come from a classifier, from CLIP, or from a text column in a table. Semantic heads like semantic sector heads map such labels into broader groups and make tables from different sources comparable.

Continual learning

When you fine-tune SAM-based pipelines or the tabular model on new domains, old skills can fade. Methods such as elastic weight consolidation, learning without forgetting and experience replay address this. Adapters offer a lightweight way to add a domain without touching the base weights.

Where the analogy stops

SAM uses spatial structure: neighboring pixels relate. Tables have no spatial grid, and row order is usually meaningless. What carries over is the recipe: broad pre-training, a prompt that specifies the task, and a lightweight head that answers quickly.

Practical guidance and best practices

Prompting

• Use a box when you can. Boxes are usually more reliable than a single point.
• Add a background point to exclude a nearby similar region.
• For ambiguous targets, request multiple masks and choose by score or by rule.
• Chain prompts: a first mask, then a correction click, then a refinement.

Quality

• Filter by predicted IoU and stability.
• Compare mask area with box area to catch obvious errors.
• Keep a human review step for high-stakes labels.
• Track the share of masks needing correction per class.

Performance

• Compute the image embedding once and reuse it for all prompts on that image.
• Use a smaller encoder variant for speed if accuracy is enough.
• Batch images where memory allows.
• Use half precision on a GPU, see mixed precision.
• For deployment on small devices, consider lighter distilled variants and quantization, see bitsandbytes for related ideas.

Domain adaptation

• Test zero-shot first on a labeled sample.
• If masks are poor, fine-tune the decoder or add adapters rather than retraining the whole encoder.
• Add domain-specific prompts such as boxes from a domain detector.
• Use active learning to choose which images to correct.

Data hygiene

• Record the model version, prompts and thresholds with each mask.
• Store masks in a compact format such as run-length encoding.
• Keep original images unmodified.

Example of a per-mask table

A status column makes the pipeline auditable. Accepted rows feed training or measurement, review rows go to a person, and no_mask rows tell downstream models that processing ran but found nothing.

A short checklist for a first project

• Test zero-shot on 50 hand-labeled images from your own domain.
• Choose the prompt type that works best, usually boxes.
• Add automatic quality filters.
• Put a person in the loop for the first thousand labels.
• Train the production model on the cleaned labels.
• Monitor mask quality in production with periodic samples.

Privacy and safety notes

Images can contain faces, license plates or other personal data. Segmentation makes it easy to cut such content out, which can be used for good, such as blurring, or for bad, such as tracking. Follow the rules that apply to your data, store only what you need and limit access.

Working with the table side

• Define a stable schema: one row per mask with class, score, area, bounding box and image id.
• Mark rejected or missing masks explicitly.
• Aggregate to per-image or per-part rows before modeling.
• Use SHAP to explain downstream predictions.

Common mistakes

• Treating SAM as a classifier. It does not output class names.
• Using a single ambiguous click and accepting the first mask without checking.
• Recomputing the image embedding for every prompt, wasting time.
• Assuming zero-shot quality in a specialized domain without testing.
• Using automatic mode and expecting clean object-level masks. It often returns parts and background regions.
• Ignoring license and usage terms of code and checkpoints.
• Training on SAM-made labels without any human check, which can bake systematic errors into a model.
• Comparing masks only by eye on easy images and ignoring thin or low-contrast cases.
• Forgetting to convert pixel areas to physical units with calibration.
• Storing only masks and not the prompts and versions that produced them.

Frequently asked questions

What is SAM in machine learning?

SAM stands for Segment Anything Model. It is a promptable segmentation foundation model that returns a mask for an object you indicate with a point, a box or a rough mask. It was trained on a very large mask dataset and works on many kinds of images without extra training.

Does SAM know what objects are called?

No. It finds the outline of an object but does not assign a class name. To get names, combine it with a detector that supplies labeled boxes, with a classifier, or with an embedding model such as CLIP that scores crops against text descriptions.

What does zero-shot mean for SAM?

Zero-shot means the model can segment objects it was not specifically trained on and needs no additional training on your data to produce a mask. Quality still varies by domain, so it is wise to test on a labeled sample from your own images.

How is SAM different from Mask R-CNN?

Mask R-CNN is trained on labeled masks for a fixed list of classes and outputs classes, boxes and masks together. SAM is prompted at use time, needs no class training, and outputs only masks. Many teams use SAM to create labels and then train a closed-set model for production.

Why does SAM return several masks for one click?

A single click can refer to a part, a whole object or a group, so the correct answer is ambiguous. SAM returns multiple plausible masks with quality scores so that you can choose the right level of detail or let a rule decide.

Is SAM fast enough for real time?

The prompt and decoder step is fast, but the image encoder is heavy and is run once per image. For interactive tools this feels responsive. For real-time video on small devices you may need lighter variants or a different model, so test on your hardware.

Can SAM segment videos?

The original model works on single images. Extensions for video propagate masks across frames, and you can also combine image SAM with a tracker. Check the current releases and test temporal stability on your footage.

Can I fine-tune SAM on my domain?

Yes. Common choices are fine-tuning the mask decoder, adding lightweight adapters, or supplying domain prompts such as boxes from a specialized detector. Start with zero-shot evaluation to see whether adaptation is necessary.

How can segmentation outputs be used with tabular models?

Each mask can be turned into numbers such as area, perimeter and shape ratios, and then aggregated per image or per part. Those columns, joined with sensor and process data, can feed tabular models for prediction, imputation or anomaly detection. Rejected masks should be stored as missing, not zero.

Do I need a GPU to run SAM?

A GPU is strongly recommended for the larger image encoders because the encoding step is heavy. Smaller or distilled variants can run on CPUs or edge devices with lower speed or accuracy, so choose according to your latency budget.

Glossary

• Ambiguity-aware prediction: returning several valid masks for an unclear prompt.
• Automatic mask generation: segmenting an image using a grid of point prompts.
• Data engine: a model-in-the-loop process that builds datasets.
• Dice loss: an overlap-based loss useful for small regions.
• Foundation model: a large model pre-trained on broad data and used through prompts or light adaptation.
• Image embedding: a feature grid describing an image.
• Image encoder: the heavy network that creates the image embedding.
• Mask decoder: the light network that turns embeddings into masks.
• Predicted IoU: the model's own estimate of mask quality.
• Prompt: a hint such as a point, box or mask.
• Prompt encoder: the part that converts prompts into vectors.
• SA-1B: the large segmentation dataset released with SAM.
• Stability score: a check on whether a mask is robust to threshold changes.
• Zero-shot: working without task-specific training.

Summary

SAM is a promptable segmentation foundation model. A heavy image encoder runs once, then a light prompt encoder and mask decoder turn clicks, boxes or rough masks into accurate masks in a blink. It was trained on a very large dataset built by a data engine, and it works zero-shot on many kinds of images. It does not name objects, struggles with very fine structures, and needs testing in specialized domains.

Pair it with YOLO or DETR for class-aware boxes, CLIP for names, and use it to label data for Mask R-CNN. For the tabular side, masks become measurable columns that models such as TabPFN and TabICL can use, and SAM's prompt-and-adapt recipe mirrors how table foundation models learn from examples in context.

Related

It belongs to Vision and multimodal in my stack, under Foundation models and applied ML. Others in the same group: