YOLO

Overview

YOLO, short for "You Only Look Once", is a family of object detectors that find and label objects in an image with a single forward pass of a neural network. Instead of proposing regions first and classifying them second, YOLO predicts bounding boxes and class probabilities directly from the whole image. This makes it fast enough for live video, which is why it is popular for production lines, traffic cameras, drones and robots.

The main trade-off is simple. YOLO gives you speed and a clean deployment story, and in return you accept some limits on very small objects, crowded scenes and pixel-level detail. If you need exact object outlines, look at Mask R-CNN or SAM. If you want a detector without anchors or hand-built post-processing, look at DETR. This page explains the idea, the history, the mechanics, a worked example, and how detection outputs can feed models that learn from tables.

What it is

YOLO is not a single model. It is a design philosophy and a lineage of models that share one idea: treat detection as one regression problem. The network looks at an image once and outputs a fixed grid of predictions. Each prediction says "there may be an object here, this is its box, and this is its class".

The core promise

• One network, one pass, one set of outputs.
• Boxes and classes are learned together with one training loss.
• Inference speed is high because there is no separate region proposal stage.
• The model can be trained end to end on labeled images.

What it outputs

For every image, a YOLO model returns a list of detections. Each detection has:

• A bounding box, usually as center x, center y, width and height.
• A confidence score that an object is present.
• A class label, or a distribution over classes.

Later versions also offer variants for instance segmentation, pose estimation and oriented boxes, but the original job is plain box detection.

Everyday analogy

Imagine a security guard who glances at a whole room once and immediately says "a chair here, a person there, a laptop on that desk". A slower guard first walks the room marking spots that look interesting, then goes back to each spot to decide what it is. The first guard is YOLO. The second guard is a two-stage detector such as Faster R-CNN. The first is quicker. The second can be more careful on hard cases.

Where it came from

Before YOLO, the leading detectors followed a two-step recipe. The R-CNN line of work, followed by Fast R-CNN and Faster R-CNN, first proposed candidate regions and then classified each one. These models were accurate but slow, because many regions needed processing.

The original paper

The first YOLO paper appeared in 2015 and was presented in 2016. It framed detection as regression from image pixels to box coordinates and class probabilities. It ran in real time on a GPU of that era and became a reference point for fast detection.

The versions, in broad strokes

• YOLOv1 introduced the grid-based single-pass idea.
• YOLOv2, also called YOLO9000, added anchor boxes, batch normalization and a joint training scheme across detection and classification labels.
• YOLOv3 (2018) used a deeper backbone and predictions at three scales, which helped with small objects.
• YOLOv4 (2020) collected many training and architecture tricks into a strong, practical recipe.
• Later releases came from several groups and open source communities. They changed backbones, necks, heads, loss functions and training pipelines. Names and numbering after the early versions are not governed by one authority, so always check which code base and license you are using.

Why the lineage matters

The history of YOLO shows a pattern that recurs across machine learning. A simple idea gets popular, then the community adds engineering improvements one by one: better feature pyramids, better augmentation, better losses, anchor-free heads, and easier export formats. When you compare "YOLO" results, you are comparing a specific version, size and training recipe, not one fixed algorithm.

How it works

This section walks through the classic mechanism, then notes how modern versions differ.

Step 1: Resize and normalize the image

The image is resized to a fixed input size, such as 640 by 640 pixels, often with padding to keep the aspect ratio. Pixel values are scaled. The network always sees the same shape, which makes batching simple.

Step 2: Extract features with a backbone

A convolutional backbone turns pixels into feature maps. Early layers detect edges and textures. Deeper layers detect parts and whole shapes. Think of it as a series of progressively more abstract summaries of the picture, like going from "dark line" to "wheel" to "car".

Step 3: Combine scales with a neck

Objects come in many sizes. A neck, often a feature pyramid style module, merges feature maps from several depths. High-resolution maps help with small objects. Low-resolution maps carry strong meaning for large ones. The neck passes information both ways so each scale gets the best of both.

Step 4: Predict on a grid with a head

The final feature maps are treated as grids. Each cell is responsible for objects whose centers fall inside it. For each cell, the head predicts:

• Box offsets and sizes relative to the cell or to a reference anchor.
• An objectness score.
• Class scores.

Classic versions used anchor boxes, which are prior shapes such as tall, wide and square. The network learns small corrections to the closest anchor. Many modern versions are anchor-free, predicting distances from a point to the box edges instead.

Step 5: Decode the predictions

Raw outputs are converted into image coordinates. Scores below a threshold are dropped. Thousands of candidate boxes may remain, and many overlap on the same object.

Step 6: Non-maximum suppression

Non-maximum suppression, or NMS, cleans up duplicates. The idea is easy to explain with a photo contest. If five photos show the same sunset, you keep the best one and discard the near copies. In detection, boxes are sorted by score. The top box is kept. Any other box that overlaps it by more than a chosen intersection over union value is removed. The process repeats.

Step 7: Return detections

The final list is mapped back to the original image size. A downstream system can draw boxes, count objects, trigger an alarm or log events.

How training works

Training needs images with labeled boxes. For each training image the loss has several parts:

• A box regression term, often based on intersection over union, which measures how well the predicted box overlaps the true box.
• An objectness or quality term, which teaches the model where objects are.
• A classification term, which teaches the label.

A key design step is label assignment, which decides which predictions are responsible for which ground-truth boxes. Older versions used a fixed rule such as "the anchor with the best shape match in the center cell". Newer ones use dynamic assignment, where the assignment adapts to the current quality of predictions. Better assignment is one of the main reasons newer versions train better.

Data augmentation

YOLO training pipelines are known for heavy augmentation. Common tricks include random scaling, flips, color jitter, mosaic (stitching four images into one), and mixing images. These force the model to see objects in varied contexts and sizes. They matter a lot when your dataset is small.

Metrics used to judge it

• Intersection over union (IoU): overlap area divided by union area of two boxes.
• Precision: of the boxes predicted, how many were right.
• Recall: of the true objects, how many were found.
• Average precision (AP): area under the precision recall curve for one class.
• Mean average precision (mAP): AP averaged over classes, sometimes over several IoU thresholds.
• Frames per second and latency: how fast inference runs on your hardware.

Reading a precision recall curve

Every threshold choice moves you along a curve. A high threshold keeps only confident boxes, so precision is high and recall is low. A low threshold keeps more boxes, so recall rises and precision falls. Think of a smoke alarm. A very sensitive alarm never misses a fire but rings when you toast bread. A less sensitive one stays quiet during toast but may be late for a real fire.

• Plot precision against recall for each class.
• Mark the point that matches your cost of a miss versus a false alarm.
• Use that point's score as the production threshold, then confirm on fresh data.

Why speed and accuracy trade off

Larger backbones see more detail but cost more compute. Larger input images keep small objects visible but cost more compute. More scales in the neck help but add layers. Each choice moves you along a speed and accuracy frontier. The art is to pick the smallest model that meets your accuracy target on your own data, then stop.

Regression targets in more detail

Boxes can be encoded in different ways. Some heads predict the center and size. Others predict four distances from a point to the left, top, right and bottom edges. The second form is natural for anchor-free designs because it needs no reference shape. Losses based on overlap, such as variants of IoU loss, care about the whole box at once rather than each number separately. That is closer to what a human means by "a good box".

Tracking across frames

Detection works frame by frame. Many applications need identity over time: this is the same part as in the last frame. Trackers link detections across frames using position, motion and appearance. A simple tracker predicts where each box should move next and matches new boxes to the nearest prediction. Tracking turns a stream of detections into a clean table of object trajectories, with one row per object per frame and a stable id.

Dealing with class imbalance

Most images contain mostly background. Within objects, some classes are rare, such as a severe defect. Common remedies include sampling more images that contain rare classes, weighting the loss, and collecting more of the rare case. Be careful with synthetic copies of a rare defect pasted onto backgrounds. If the paste edges look fake, the model may learn the edge instead of the defect.

Key concepts and terms

A worked example

Suppose a factory wants to find scratches and dents on metal parts moving on a conveyor. A camera takes frames at a steady rate. The goal is to flag a part when a defect appears.

Step 1: Collect and label

Collect a few thousand images that include normal parts and defective parts. Annotators draw boxes around each defect and assign a class: "scratch", "dent", "stain". Keep a separate validation set from different shifts or days, so you test generalization rather than memorization.

Step 2: Choose a model size

Most YOLO releases come in several sizes, from tiny to extra large. A small model runs on an edge device. A large model gives better accuracy on a server GPU. Start with a small or medium one and see if accuracy is enough.

Step 3: Train

A typical training call using a popular open source package looks like this. The package and weights file names are placeholders for whichever release you use.

from ultralytics import YOLO

model = YOLO("yolo-small.pt")  # placeholder weights name
model.train(
    data="defects.yaml",
    epochs=100,
    imgsz=640,
    batch=16,
)
metrics = model.val()
print(metrics.box.map)  # mAP over IoU thresholds

The data file describes where images live and the class names:

path: /data/defects
train: images/train
val: images/val
names:
  0: scratch
  1: dent
  2: stain

Step 4: Run inference

results = model.predict("frame_0001.jpg", conf=0.35, iou=0.5)
for r in results:
    for box in r.boxes:
        cls = int(box.cls)
        score = float(box.conf)
        x1, y1, x2, y2 = box.xyxy[0].tolist()
        print(cls, round(score, 2), x1, y1, x2, y2)

The conf value is the confidence threshold. The iou value is the NMS overlap threshold.

Step 5: A small numeric walk-through of NMS

Imagine three boxes for one scratch, with scores 0.9, 0.8 and 0.4.

• Box A, score 0.9, is kept first.
• Box B, score 0.8, overlaps A with IoU 0.7. With an NMS threshold of 0.5 it is removed.
• Box C, score 0.4, overlaps A with IoU 0.1. It stays, because it may be a different scratch.

The result is two detections instead of three.

Step 6: Turn detections into decisions

A plant system should not react to single frames. A simple rule: raise an alarm if a defect class is detected with a score above 0.5 in at least three of the last five frames. That reduces false alarms from noise.

Step 7: Export for deployment

Export to a portable format and run with an optimized runtime. Many teams also quantize to lower precision. Check accuracy after export, because small changes in numeric precision can shift scores near thresholds. Libraries such as OpenCV can load exported models and handle camera capture and drawing.

Step 8: Monitor in production

Log counts of detections per class, average confidence, and the share of frames with any detection. A sudden shift often means a lighting change, a dirty lens or a new material, not a real change in defects.

Strengths and limitations

Strengths

• Very fast inference, suitable for live video and edge devices.
• Simple pipeline: image in, boxes out.
• Many sizes let you trade accuracy for speed.
• Large community, many tutorials, and easy export tooling.
• Strong results when you have enough labeled data of your own domain.
• Works well on medium and large objects with clear boundaries.

Limitations

• Small objects that occupy few pixels are hard, especially in high-resolution scenes.
• Dense crowds with heavy overlap can cause missed objects or merged boxes.
• Boxes are coarse. They do not give exact outlines. For that, add a segmentation head or use a dedicated segmenter.
• NMS has tunable thresholds that interact with scene density.
• Performance depends on labeling quality. Inconsistent boxes teach the model inconsistent habits.
• Versions differ in license terms and code quality. Check before commercial use.
• Domain shift hurts. A model trained in one factory may fail in another with different lighting.

Honest remarks on benchmarks

Published numbers on public detection sets are useful for rough ranking but they do not predict your result. Your images, classes, camera angle and label style decide the outcome. Always test on a held-out set that looks like production.

How it compares with alternatives

A common combined pattern

Teams often chain tools. A fast detector finds candidate boxes. A segmenter such as SAM refines each box into a mask. An embedding model such as CLIP checks or relabels the crop. Each tool does what it is best at.

When YOLO is the right pick

• You need real-time speed.
• You can label your own data.
• Boxes are enough for the decision.
• You will deploy to a modest GPU or an edge device.

When another tool fits better

• You need exact shapes: use an instance segmentation model.
• You have no labels at all: consider a zero-shot approach with CLIP or a promptable model.
• You want a simpler training graph with no hand-built post-processing: consider DETR and its faster-converging successors.

Role in tabular foundation models

YOLO is a vision model, so it does not read tables. Still, it connects to models that learn from raw tables in several concrete ways. See the pages on research, architecture and benchmarks for how a table-native model is built and judged.

Detection output is tabular data

Every detection is a row: image id, timestamp, class, score, x, y, width, height. Over a day on a production line this becomes a large table with a mix of types. Class is categorical. Score and sizes are numeric. Time is ordered. Some fields are missing when a frame is dropped. A table-native model can learn from this directly, without hand-crafted features, and without caring whether the columns have names.

Fusing vision with sensor columns

Factories rarely rely on cameras alone. They also log temperature, vibration, line speed, batch id and tool wear. A common design is to run a detector, aggregate its outputs into per-part columns such as "number of scratch detections" and "largest defect area", then join them with sensor columns. A tabular model like TabPFN, TabICL, TabSTAR or FT-Transformer can then predict scrap risk, remaining tool life or the cause of a defect. In-context learning is attractive here because each plant has only a few hundred labeled parts, so a model that adapts from a small table of examples without retraining is practical.

Missing values and imputation

Detector outputs have natural gaps. A camera may be offline, a part may be occluded, or a frame may be skipped. Treating these gaps as missing cells and filling them with a learned approach, as described under masked cell modeling and imputation networks, is often better than filling with zero. Zero would falsely say "no defects".

Anomaly detection

A detector trained on known defects cannot see unknown ones. A table-based anomaly detector can catch drift in the statistics of detections and sensors, for example a rising average defect area or a change in the ratio of classes. Methods such as isolation forest, one-class SVM and general anomaly detection apply here. A tabular foundation model can do the same with in-context examples of normal rows.

Semantic and sector labeling

Detection class names such as "scratch" or "forklift" are text. Table-native models that handle text columns, and models with semantic sector heads, can map those labels and other free text into higher-level categories like "surface defect" or "material handling". This helps unify label sets across plants.

Continual learning

A detector gets retrained when new defect types appear. The table of downstream decisions must adapt too. Methods such as elastic weight consolidation, learning without forgetting and experience replay help a model learn new patterns without forgetting old ones. These ideas apply on the vision side and on the tabular side.

Pre-training on synthetic data

Real defect data is rare. Teams often generate synthetic images or synthetic rows. For tables, synthetic data generation is also how some foundation models are pre-trained: they see many generated tables with different structures and learn general prediction behavior. The same intuition, that variety in training worlds builds flexible models, shows up in vision when detectors are trained with strong augmentation.

Example: a table built from detections

Here is what a small slice of a detection log might look like once stored as a table.

Rolling this up to one row per part gives columns like count of detections, maximum score, total defect area and the dominant class. Those columns join with sensor readings. The result is an ordinary mixed-type table, which is exactly the shape that tabular foundation models are built to read.

Why this matters for in-context learning

Suppose a new plant starts with only 200 labeled parts. Training a deep model from scratch on 200 rows is risky. A model that was pre-trained on many varied tables can take those 200 labeled rows as context and predict the label of a new row in one pass. No gradient steps are needed on the new plant. That fits the way detection feeds decisions: the detector is trained once on pixels, the tabular model adapts quickly on rows.

Practical rollup features

• Count of detections per class in the last N frames.
• Maximum and mean confidence per part.
• Total and largest defect area as a share of the part.
• Time since the last detection.
• Share of frames with any detection in a shift.
• Change of the above compared with the previous shift.

These simple columns carry a surprising amount of signal, and they are easy to explain to operators, which helps adoption.

A note on explainability

Operators trust a system more when it can show why. A drawn box on the image is a form of explanation for the detector. For the table model on top, attribution methods such as SHAP can show which columns pushed a risk score up. Together they give a story: "the scratch area was large and the line speed was high".

Limits of the analogy

YOLO and tabular foundation models solve different problems. YOLO exploits spatial structure in pixels. A tabular model has no spatial grid; rows are exchangeable and columns have mixed types and no fixed order. What they share is the engineering mindset of a fast, general, single-pass predictor that you can put into a pipeline.

Practical guidance and best practices

Data

• Label consistently. Write annotation rules and review samples.
• Include hard negatives: images with no objects, or with look-alikes.
• Cover the real range of lighting, angle, background and motion blur.
• Balance classes or use class weights if one class dominates.
• Split by time or by source, not by random frame, to avoid leakage from near-identical frames.

Training

• Start from pre-trained weights and fine-tune. This usually beats training from scratch.
• Choose an input size that keeps small objects visible. Larger input costs speed.
• Watch both training and validation curves. If validation stalls while training improves, you are overfitting.
• Keep augmentation realistic. Do not flip images if direction matters, for example text or a left-right defect meaning.
• Use mixed precision for speed, see mixed precision.
• Use active learning to pick the next images to label: those where the model is unsure.

Evaluation

• Report precision and recall per class, not only mAP.
• Pick the confidence threshold based on cost. A missed defect and a false alarm rarely cost the same.
• Check performance by object size and by condition (day, night, dirty lens).
• Look at errors visually. A hundred wrong predictions teach more than a single number.

Deployment

• Benchmark on the target hardware with the real input pipeline, including decoding and drawing.
• Use batching on servers and single-frame processing on edge devices.
• Consider quantization and export to a portable format, then re-validate accuracy. See also bitsandbytes for quantization ideas in other model families.
• Add temporal smoothing or tracking so one noisy frame does not trigger action.
• Keep a record of model version, thresholds and training data version for audits.

Monitoring

• Track the distribution of scores and classes over time.
• Sample frames for human review regularly.
• Retrain on a schedule or when drift alarms fire.

Edge cases worth testing

• Empty frames with only background.
• Two objects touching or overlapping.
• Objects cut by the image border.
• Strong reflections or glare on shiny metal.
• Motion blur when the line speeds up.
• A new supplier whose parts have a different finish.

Add a few labeled examples of each to the validation set, so a regression in one of them shows up before a customer notices.

A short checklist before launch

• The validation set comes from a different time or source than training.
• The chosen threshold is justified by cost, not by default.
• Latency was measured end to end, not only the network.
• A fallback exists when the camera or model fails.
• Version, weights and data are recorded.
• Someone owns retraining and monitoring.

Data quality for the table side

If detections will feed downstream tabular models, store them cleanly. Keep one row per detection with a stable schema. Record the model version and threshold used. Keep timestamps in one time zone. Mark dropped frames explicitly rather than leaving silent gaps. A tabular foundation model can cope with missing cells, but it cannot recover information you never recorded.

Cost and energy

A model that runs on a small device uses far less energy than one that streams video to a server. Estimate cost per thousand frames for each option. Often an edge model with occasional server review is the cheapest design that still meets the accuracy goal.

Common mistakes

• Judging a model by mAP on a public set and assuming it will hold on your images.
• Random train and validation splits over video frames, which leak near-duplicates and inflate scores.
• Using a very low confidence threshold in production, then drowning operators in false alarms.
• Ignoring label noise. Overlapping or loose boxes cap achievable accuracy.
• Forgetting to apply the same resize and padding at inference as in training.
• Comparing speed numbers measured on different hardware, batch sizes or precisions.
• Treating NMS settings as fixed. Dense scenes may need a higher overlap threshold, sparse scenes a lower one.
• Skipping the background class examples, which leads to many false positives on empty frames.
• Not checking the license of the code and weights before shipping a product.
• Retraining with new data and silently losing performance on old classes.

Frequently asked questions

What does YOLO stand for?

YOLO stands for "You Only Look Once". The name describes the central idea that the network examines the whole image in a single forward pass and outputs all boxes and classes at once, instead of running a separate classifier on many proposed regions.

Is YOLO a single model or a family?

It is a family. The first version appeared in 2015, and many later versions from different groups changed the backbone, neck, head, losses and training recipe. When someone says YOLO, ask which version, which size and which code base, because results and licenses differ.

Is YOLO more accurate than Mask R-CNN or DETR?

It depends on the task and the version. YOLO is usually chosen for speed and simple deployment, while two-stage and transformer detectors may do better in some settings, especially when exact masks or end-to-end set prediction matter. Always compare on your own held-out data.

Can YOLO detect objects it has never seen?

Not by itself. A standard YOLO model only detects the classes it was trained on. For new classes you need labeled examples and fine-tuning, or you can combine it with a zero-shot model such as CLIP or a promptable segmenter to handle open-ended labels.

How much data do I need to fine-tune YOLO?

There is no fixed number. With pre-trained weights, a few hundred well-labeled images per class can give a useful first model, and thousands usually give a stable one. Label quality, variety of conditions and class balance matter as much as raw count.

Does YOLO work on small objects?

It can, but small objects are its known weak spot. Using a larger input size, a model with multi-scale predictions, tiling a large image into crops, and including enough small-object examples in training all help. Expect to tune and test this carefully.

What is non-maximum suppression and can I avoid it?

NMS removes duplicate boxes that overlap on the same object by keeping the highest-scoring one. Classic YOLO needs it. Some newer designs and transformer detectors such as DETR are built to avoid it by predicting a set of unique objects directly.

Can YOLO output masks instead of boxes?

Several YOLO variants add a segmentation head and return instance masks, and you can also pass YOLO boxes to a promptable segmenter like SAM to get masks. For the highest mask quality on tight shapes, test a dedicated instance segmentation model as well.

How do detections connect to tabular machine learning?

Each detection is a row with a class, a score and box geometry. Aggregated per image, per part or per time window, detections become columns that can be joined with sensor and process data and fed to tabular models for prediction, imputation or anomaly detection. This is a common way to turn pixels into decisions.

Can I run YOLO on an edge device?

Yes. Small and tiny sizes are designed for that. Export the model to a portable format, apply quantization if needed, and measure accuracy and latency on the actual device with your real camera pipeline before you commit to a design.

Glossary

• Anchor box: a preset box shape that a detector adjusts to fit an object.
• Anchor-free: a detector design that predicts box edges directly without preset shapes.
• AP and mAP: average precision for a class, and its mean over classes.
• Backbone: the network part that extracts features from pixels.
• Bounding box: a rectangle that marks where an object is.
• Confidence score: the model's estimate that a detection is correct.
• Feature pyramid: a structure that combines features from several scales.
• Head: the final layers that produce predictions.
• IoU: intersection over union, a measure of box overlap.
• Label assignment: the rule that links predictions to ground-truth objects in training.
• Mosaic augmentation: joining several training images into one sample.
• NMS: non-maximum suppression, removal of duplicate boxes.
• Neck: the module between backbone and head that merges scales.
• Objectness: a score for whether any object is present at a location.
• Quantization: storing weights and activations with fewer bits to speed up inference.
• Single-stage detector: a detector that predicts boxes directly without a proposal step.

Summary

YOLO is a family of fast, single-pass object detectors. It looks at the whole image once and returns boxes, scores and classes, which makes it a strong choice for video, edge devices and production lines. Its strengths are speed, simplicity and a rich ecosystem. Its weaknesses are small objects, crowded scenes and coarse box shapes.

For related vision tools, see Mask R-CNN, DETR, SAM and CLIP. For the tabular side, detections become rows and columns that models like TabPFN and TabICL can learn from, with imputation, anomaly detection and continual learning handling gaps, drift and change. Test on data that looks like production, set thresholds by cost, and monitor after launch.

Related

It belongs to Vision and multimodal in my stack, under Foundation models and applied ML. Others in the same group: