Mask R-CNN

Overview

Mask R-CNN is a neural network that finds every object in an image and draws a pixel-accurate mask around each one. It extends Faster R-CNN, a two-stage object detector, by adding a third output: a small mask branch that predicts which pixels inside each detected box belong to the object. The result is instance segmentation, which means separating individual objects even when they are of the same class, such as three different scratches on one part.

Use Mask R-CNN when a rectangle is not enough. If you need to measure the area of a defect, trace the outline of a cell, or cut one object out of a scene, masks give you real shapes. If you only need fast boxes, YOLO is usually simpler. If you want a zero-shot, promptable segmenter, look at SAM. This page covers the idea, the history, the mechanics, a worked example and how the outputs can feed models that learn from tables.

What it is

Mask R-CNN is a two-stage, region-based convolutional network. The name breaks down as "mask" for the pixel-level output and "R-CNN" for region-based convolutional neural network.

The three outputs

For every object it finds, the model returns:

• A class label, such as "scratch" or "person".
• A bounding box, refined from a rough proposal.
• A binary mask, a small grid that marks object pixels as one and background as zero.

What makes it different from detection

A detector says "there is a dent somewhere in this rectangle". Mask R-CNN says "these exact pixels are the dent". The box includes background corners. The mask does not. That difference matters when you compute areas, lengths or overlaps.

Instance versus semantic segmentation

• Semantic segmentation labels each pixel with a class but does not separate individual objects. All "car" pixels share one label.
• Instance segmentation separates each object. Two cars next to each other get two different masks.
• Mask R-CNN does instance segmentation. Panoptic segmentation combines both views and is a later extension of the same family of ideas.

Everyday analogy

Picture a coloring book. A detector draws a box around each animal. Mask R-CNN colors inside the lines of each animal, using a different crayon for each one, even if two dogs overlap. The box is the sketch. The mask is the colored picture.

Where it came from

The story runs through a sequence of papers from the same line of research.

R-CNN

R-CNN (2014) showed that features from a convolutional network, applied to candidate regions found by a classic proposal algorithm, gave much better detection than older hand-built features. It was accurate but slow, because each region was processed separately.

Fast R-CNN

Fast R-CNN (2015) shared computation. The whole image went through the network once, and a region pooling step cut out features for each candidate region. Training became a single multi-task process.

Faster R-CNN

Faster R-CNN (2015) replaced the external proposal step with a learned region proposal network that shares features with the detector. The pipeline became one network trained end to end. It became a standard baseline for years.

Mask R-CNN

Mask R-CNN was published in 2017 and presented at a major computer vision conference that year. It added a mask branch in parallel with the box branch, and introduced RoIAlign, a more precise way to extract region features. It won a best paper award at that conference and became the default reference for instance segmentation.

Later influence

The design inspired many follow-ups: cascade versions with several refinement stages, versions with stronger backbones such as transformer backbones, and tools that package the model for easy training. Newer promptable and transformer-based segmenters such as SAM and DETR-style models have changed the field, but Mask R-CNN remains a clear, teachable and widely used baseline.

How it works

The pipeline has a backbone, a proposal stage, a region feature extractor, and several heads. Each part is explained below with an everyday comparison.

Step 1: Backbone feature extraction

The image goes through a convolutional backbone, often a residual network. Early layers find edges and textures. Later layers find parts and shapes. The output is a set of feature maps, which are like a stack of summaries of the picture at different levels of detail.

Step 2: Feature pyramid

Objects come in many sizes. A feature pyramid network builds maps at several scales and passes strong meaning from coarse layers to fine layers. Think of looking at a city with a map that has zoom levels. Small objects are best seen at a fine zoom. Large objects are best seen at a coarse zoom. The pyramid gives you every zoom, each with good meaning.

Step 3: Region proposal network

The region proposal network slides over the feature maps and, at each location, scores a set of anchor boxes of different sizes and shapes. For each anchor it predicts:

• An objectness score: is there any object here?
• A box adjustment: how should the anchor be nudged to fit?

The top-scoring proposals are kept, and overlapping ones are pruned using non-maximum suppression. This is like a friend who quickly circles places on a photo that look like something, without yet saying what they are.

Step 4: RoIAlign

Each proposal has a different size, but the later heads need a fixed-size input. The older approach, region of interest pooling, rounded box coordinates to whole feature cells. That rounding shifts the box slightly. A shift of a few pixels barely hurts a classifier but badly hurts a mask.

RoIAlign removes the rounding. It samples the feature map at exact fractional positions using bilinear interpolation and then pools to a fixed grid. Analogy: instead of snapping a photo crop to the nearest grid line, you cut exactly where you meant to. This small change gave a large boost to mask quality.

Step 5: Box head

The fixed-size region features go to a head that predicts a class and a refined box for each proposal. This is the same idea as in Faster R-CNN. Proposals that do not match any object are labeled as background.

Step 6: Mask head

In parallel, a small fully convolutional network takes the same region features and predicts a mask for each class. The mask is a low-resolution grid, commonly 28 by 28 values, for each possible class. Each value is a probability that the pixel belongs to the object.

Two design details matter:

• Masks are predicted per class, and only the mask for the predicted class is used. This decouples mask prediction from classification, so the classes do not compete at the pixel level.
• Each pixel uses a sigmoid and a binary loss, not a softmax across classes.

Step 7: Paste masks back

At inference, the small mask is resized to the size of its box and placed into the full image at the box location. A threshold, often 0.5, turns probabilities into a binary mask. The result is a full-resolution mask for each object.

The training loss

Training minimizes a sum of three terms for every sampled region:

• Classification loss for the label.
• Box regression loss for the box.
• Mask loss, a per-pixel binary cross entropy, computed only on the ground-truth class mask.

Regions are labeled positive if they overlap a ground-truth object enough, measured by intersection over union. Typical settings sample a fixed number of regions per image with a set ratio of positives and negatives.

Why two stages?

The first stage produces a modest number of good candidates. The second stage then spends heavy computation on only those candidates. That concentrates effort where it matters, which is why two-stage detectors tend to be accurate on hard cases, at the cost of speed.

Sampling regions during training

Training does not use every proposal. A fixed number of regions per image is sampled, with a chosen share of positives. Positive regions overlap a true object enough. Negative regions are mostly background. If you use too few positives the mask head sees little signal. If you use too many easy negatives the box head learns to say "background" and stays lazy.

Reading the loss terms

• A high classification loss means the model confuses classes or background.
• A high box loss means the boxes are loose.
• A high mask loss means the outlines are wrong even if boxes look fine.
• A high objectness loss means the proposal stage is weak, often a sign of missing or inconsistent labels.

Watching each term separately tells you which part of the pipeline to fix.

Extra heads

Because the design is modular, other heads can share the same region features. A keypoint head can predict landmark positions, such as joints of a person. A different head can predict a quality score for each mask. The shared backbone means these tasks help each other and cost little extra.

Scale and resolution

Images are usually resized so the shorter side has a fixed length. If your objects are small, a larger resize helps but costs compute and memory. Tiling a very large image into overlapping crops, running the model on each, and merging results is a common remedy. Be careful at tile borders: an object cut by a border can be found twice, so merge masks that overlap strongly.

Failure modes to look for

• Merged instances: two touching spots become one mask.
• Split instances: one long spot becomes two masks.
• Halo errors: the mask includes a ring of background around the object.
• Missed faint objects: low contrast spots score below the threshold.
• Duplicate masks: the same object appears twice with different scores.

Each failure has a different fix. Merged and split instances point to label rules or to the NMS setting. Halo errors point to loose labels. Missed faint objects point to augmentation and threshold choices. Duplicates point to NMS thresholds.

Why the mask head is class specific

Predicting one mask per class lets the head say "if this is a dent, here is its shape" and "if this is a scratch, here is its shape" without mixing them. At inference only the mask for the predicted class is used. It is similar to having one stencil per object type and choosing the stencil after you decide what the object is.

Training data

You need images with an outline per object, stored as polygons or masks. Labeling is more costly than drawing boxes. Many teams use assisted tools that propose outlines, which a human then corrects. A promptable model such as SAM is often used for this assistance.

Evaluation

• Box AP and mask AP both appear in reports.
• Mask IoU compares predicted and true pixel sets, rather than boxes.
• AP is computed over several IoU thresholds, and sometimes split by small, medium and large objects.
• For measurement tasks, also report the error in the quantity you care about, such as area in square millimeters.

Key concepts and terms

A worked example

Goal: measure the area of corrosion spots on pipe sections from photos, so inspectors can rank pipes by damage.

Step 1: Prepare the data

Collect photos. Label each corrosion spot with a polygon. Store annotations in a common format that lists images, categories and per-object outlines. Hold out a validation set from different pipes and days.

Step 2: Fine-tune a pre-trained model

Many libraries provide a pre-trained Mask R-CNN. The example below uses the torchvision implementation, which is widely available. Weights names and details can differ by library version.

import torch
import torchvision
from torchvision.models.detection import maskrcnn_resnet50_fpn
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from torchvision.models.detection.mask_rcnn import MaskRCNNPredictor

num_classes = 2  # background + corrosion

model = maskrcnn_resnet50_fpn(weights="DEFAULT")

# replace the box predictor
in_feat = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_feat, num_classes)

# replace the mask predictor
in_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels
model.roi_heads.mask_predictor = MaskRCNNPredictor(in_mask, 256, num_classes)

params = [p for p in model.parameters() if p.requires_grad]
opt = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005)

Step 3: Training loop

model.train()
for images, targets in loader:
    # targets: list of dicts with boxes, labels, masks
    losses = model(images, targets)
    loss = sum(losses.values())
    opt.zero_grad()
    loss.backward()
    opt.step()

The model returns a dictionary of losses in training mode, including classification, box regression, mask, objectness and proposal box terms.

Step 4: Inference and measuring area

model.eval()
with torch.no_grad():
    out = model([image_tensor])[0]

keep = out["scores"] > 0.6
masks = out["masks"][keep] > 0.5        # (N, 1, H, W) booleans
pixel_areas = masks.sum(dim=(1, 2, 3))  # pixels per spot

mm_per_pixel = 0.2                       # from camera calibration
areas_mm2 = pixel_areas * (mm_per_pixel ** 2)
print(areas_mm2.tolist())

Step 5: Numeric walk-through

Suppose the camera is calibrated so that one pixel equals 0.2 millimeters. A mask covers 1,500 pixels.

• Area in square millimeters is 1,500 times 0.2 times 0.2, which is 60.
• If the box around it is 50 by 50 pixels, the box area is 2,500 pixels, or 100 square millimeters.
• The box would overestimate the damage by 40 percent. The mask gives the better number.

Step 6: Check by eye and by number

Overlay masks on a sample of validation images. Look for masks that bleed into the background, split one spot into two, or merge neighbors. Then compare predicted area with a hand-measured area on a subset, and report the typical error.

Step 7: Turn masks into table rows

Per image or per pipe section, store one row per spot with class, score, area, perimeter, centroid and a compactness measure. Roll up to per-section columns such as total damaged share and number of spots. Join them with inspection history and material data. This gives a clean table for downstream prediction of repair priority.

Strengths and limitations

Strengths

• Accurate pixel-level masks for individual objects.
• Strong, well-understood baseline with plentiful implementations.
• Handles objects of varied size thanks to the feature pyramid.
• Modular: you can swap the backbone, add keypoint or other heads.
• Good for measurement tasks where area and shape matter.
• Pre-trained weights make fine-tuning feasible with moderate data.

Limitations

• Slower than single-stage detectors such as YOLO. Real-time use needs a strong GPU or a lighter variant.
• Mask resolution is low by default, so thin structures and fine edges can look blobby.
• Needs polygon or mask labels, which are costly to produce.
• Many hyperparameters: anchors, sampling ratios, thresholds.
• Crowded scenes and heavy occlusion are still hard.
• Fixed set of classes. New classes need labeled data and training.
• Relies on NMS and anchor design, which newer end-to-end methods such as DETR aim to remove.

When quality suffers

Mask quality drops for very thin objects such as cracks or hairlines, because a 28 by 28 grid cannot represent a one-pixel-wide line well. For those, consider a dedicated segmentation network that works at full resolution, or refine the mask with a higher-resolution step.

How it compares with alternatives

Common combined patterns

• Use a promptable model to speed up mask labeling, then train Mask R-CNN on the labels for a closed set of classes.
• Use a fast detector to find regions, then a segmenter on the crops.
• Use an embedding model to verify or relabel each cropped mask.

Choosing quickly

• Need real-time boxes: choose a single-stage detector.
• Need exact shapes for a fixed set of classes: Mask R-CNN is a good first try.
• Need masks for arbitrary things with no training: a promptable model.
• Need long-tail labels with no boxes: combine a zero-shot embedding model with a segmenter.

Role in tabular foundation models

Mask R-CNN is a vision model and does not read tables. It still connects to table-native models in practical ways. The pages on research, architecture and benchmarks describe how a model that learns from raw tables is built and tested.

Masks become measurable columns

A mask is a rich object, but decisions rarely use it directly. Decisions use numbers: area, perimeter, elongation, mean color inside the mask, distance to an edge. Each becomes a column. A single image yields a variable number of objects, so teams aggregate to per-image or per-part rows. The final table has mixed types: numeric measures, a categorical class, an ordered timestamp, and many missing cells when nothing was found or the camera failed.

Mixed types, no column names

A table-native foundation model must work when columns have odd names or none at all, and when types are mixed. Mask-derived features are a good test of that. A column called "f17" might be area in square millimeters or a count. The model has to infer from values and context. Models such as TabPFN, TabICL, TabSTAR, SAINT and FT-Transformer approach this in different ways, from in-context learning to learned column embeddings.

In-context learning on small inspection sets

Inspection data is usually small: a few hundred labeled pipes or parts. A pre-trained tabular model can take those rows as context and predict repair priority for a new row without any retraining. That fits the Mask R-CNN workflow well. The vision model is trained once on pixels. The tabular model adapts on the spot to each site.

Imputation

When a pipe photo is missing or blurry, its mask features are missing. Filling them with the mean hides the fact that nothing was measured. Learned imputation, as described under masked cell modeling and imputation networks, predicts plausible values from other columns and keeps an indicator of what was filled. Missingness itself can carry meaning, for example when blurry images occur more on hot days.

Anomaly detection

Mask statistics drift when a process changes. A shift in the distribution of spot areas or in the number of spots per section is a signal. Methods such as isolation forest and one-class SVM work on those rolled-up columns. See also the general page on anomaly detection.

Sector and semantic classification

Labels from vision models are words. Table-native systems with semantic sector heads can map such text into broader categories, for example "corrosion" and "rust bloom" into one group. This helps combine data from sites that use different label vocabularies.

Continual learning

New defect types appear. When the vision model is updated, the meaning and range of its output columns can shift. The tabular model must adapt without forgetting. Techniques like elastic weight consolidation, learning without forgetting and experience replay address this problem in both settings.

Pre-training on synthetic data

Real labeled inspection data is scarce. Tabular foundation models are often pre-trained on large collections of generated tables with varied structure, then used on real data without further training. See synthetic data generation. On the vision side, synthetic renders and pasted objects play a similar role in training segmenters.

Where the analogy stops

Mask R-CNN uses spatial structure and local patterns. Tables have no spatial grid, and row order is not meaningful. Still, a few ideas travel well: a set of candidate regions resembles a set of rows, and a head that predicts a value per region resembles a head that predicts a value per cell. Both reward careful handling of padding, missing items and variable counts.

Practical guidance and best practices

Data and labels

• Write clear labeling rules, including how to treat partly visible objects and overlaps.
• Draw tight outlines. Loose polygons teach the model to be loose.
• Include images with no target objects.
• Split by source, day or site to avoid near-duplicate leakage.
• Use assisted labeling to cut effort, but review the results.

Training

• Fine-tune from pre-trained weights. Training from scratch needs far more data.
• Freeze early backbone layers at first if the dataset is small.
• Use data augmentation that keeps masks aligned: flips, scale jitter, color changes. Apply the same geometric transform to image and masks.
• Tune the learning rate with a warm-up and a gradual decay.
• Use mixed precision to speed training and save memory.
• Consider active learning to choose which images to label next.

Inference

• Pick the score threshold from your cost trade-off, not from a default.
• Tune the number of proposals kept. More proposals raise recall and cost.
• Resize images so objects are not tiny after downscaling.
• Convert the mask threshold into a design choice and test its effect on area estimates.

Measurement quality

• Calibrate pixel size, and correct lens distortion if the camera needs it.
• Keep a physical reference object in some images to check the scale.
• Report uncertainty on areas, not just a single number.

Deployment

• Measure latency on target hardware.
• Export to a portable format and test outputs match the original.
• Consider lighter backbones or distilled models for edge use.
• Log model version, thresholds and calibration with each result.

A small checklist for a first project

• Define the one quantity you need to output, such as area or count.
• Label a pilot set of 100 images and train a quick model.
• Look at errors and fix label rules before labeling more.
• Add images from every lighting and surface condition.
• Freeze the validation set and never train on it.
• Decide thresholds from costs and write them down.

Example of a per-object table

The last row shows an image with no detection. Keep such rows, so downstream models know the image was processed and nothing was found, which differs from an image that was never processed.

Quality control for masks

• Sample masks weekly and score them by eye.
• Track mean mask area per class over time.
• Compare against a hand-measured subset to track bias.
• Alert on sudden changes in the share of images with zero detections.

Working with the table side

• Keep a stable schema for per-object rows.
• Mark missing images explicitly.
• Store raw masks or their compressed form for audit, and derived measures for modeling.
• Feed derived columns to tabular models, and explain predictions with attribution tools such as SHAP.

Common mistakes

• Using box area as if it were object area.
• Forgetting that the mask is predicted at low resolution and then resized, which blurs edges.
• Applying flips to the image but not to the masks during augmentation.
• Evaluating only box AP when the real goal is mask quality.
• Skipping camera calibration and reporting areas in pixels as if they were physical units.
• Treating the score threshold as universal. It changes with the lighting and the class.
• Letting near-identical frames land in both training and validation sets.
• Ignoring annotation noise. Inconsistent outlines cap the achievable mask AP.
• Training with too few negative images, which causes false positives on clean surfaces.
• Forgetting to retrain or recalibrate after a camera or lens change.

Frequently asked questions

What is the difference between Mask R-CNN and Faster R-CNN?

Faster R-CNN predicts a class and a bounding box for each object. Mask R-CNN keeps those outputs and adds a third branch that predicts a pixel mask for each object. It also replaces region pooling with RoIAlign so that the extracted features line up precisely with the image, which is important for good masks.

What does RoIAlign do and why does it matter?

RoIAlign extracts a fixed-size feature grid for each region without rounding the region coordinates. It samples the feature map at exact fractional positions with bilinear interpolation. This avoids small misalignments that barely affect classification but noticeably hurt mask accuracy, so it was a key part of the improvement.

Is Mask R-CNN still relevant now that promptable segmenters exist?

Yes, for closed-set tasks where you have labels and want a fixed, repeatable model with known behavior. Promptable models are excellent for zero-shot masks and for speeding up labeling, but they do not assign your own class names by themselves. Many teams use both.

How much labeled data does Mask R-CNN need?

There is no single number. With pre-trained weights, a few hundred well-labeled images can give a usable model, and a few thousand give a more stable one. Label quality, variety and the difficulty of the shapes matter as much as the count.

Can Mask R-CNN run in real time?

Standard versions are slower than single-stage detectors, and real-time use depends on the backbone, input size and hardware. Lighter backbones, smaller inputs, fewer proposals and optimized runtimes can make it fast enough for some applications. Measure on your own device.

Why are the masks only 28 by 28?

A small mask grid keeps the mask head cheap and works well for most objects. The small mask is then resized to the box size, which is why edges can look smooth or blobby. For thin or highly detailed shapes, use a higher resolution mask head or a different segmenter.

Does Mask R-CNN work for medical or scientific images?

It is used for tasks such as counting and outlining cells or finding lesions, often after fine-tuning on domain data. Domain differences are large, so check results carefully with domain experts and expect to adapt the training recipe and augmentations.

How do I use Mask R-CNN output in tabular machine learning?

Convert each mask into numbers such as area, perimeter, shape ratios and mean intensity, then aggregate per image or per part. Join those columns with other data like sensor readings and use a tabular model to predict outcomes. Missing images should appear as missing cells, not zeros.

Can it detect objects not in its training classes?

No. It predicts only the classes it was trained on. To handle new classes you can fine-tune with new labels, or combine it with a zero-shot model that matches crops to text descriptions.

What is the best backbone to use?

There is no universal best. Residual networks with a feature pyramid are a strong default. Larger or transformer-based backbones can raise accuracy at higher cost. Choose the smallest backbone that meets your accuracy and latency targets on your own validation set.

Glossary

• Anchor: a preset box shape that the proposal network adjusts.
• Backbone: the convolutional network that extracts features.
• Binary mask: a grid where each pixel is object or background.
• Box head: the branch that classifies a region and refines its box.
• Feature pyramid network: a multi-scale feature structure.
• Instance segmentation: segmentation that separates individual objects.
• IoU: intersection over union, a measure of overlap.
• Mask AP: average precision using mask overlap.
• Mask head: the branch that predicts masks for regions.
• NMS: non-maximum suppression, removal of duplicate boxes.
• Region proposal network: a network that proposes candidate boxes.
• RoIAlign: precise extraction of region features without rounding.
• Semantic segmentation: pixel labeling without separating instances.
• Two-stage detector: a detector with a proposal stage and a refinement stage.

Summary

Mask R-CNN extends Faster R-CNN with a mask branch and RoIAlign to deliver class labels, boxes and pixel-accurate masks for every object in an image. It is accurate, modular and widely supported, and it remains a go-to baseline for instance segmentation and measurement tasks. Its costs are speed, labeling effort and a low-resolution mask grid.

Compare it with YOLO for speed, DETR for end-to-end detection, SAM for promptable masks and CLIP for zero-shot labels. On the tabular side, mask-derived measurements become columns that models like TabPFN and TabICL can use, with imputation, anomaly detection and continual learning covering gaps, drift and change.

Related

It belongs to Vision and multimodal in my stack, under Foundation models and applied ML. Others in the same group: