OpenCV

Overview

OpenCV is an open source library of image processing and computer vision functions. It is written in C++ and has bindings for Python, Java and other languages. With a few lines of code you can read an image, resize it, blur it, detect edges, find contours, match features, track motion, calibrate a camera and run neural network models. It is one of the most widely used tools in vision, from student projects to factory inspection lines.

OpenCV is best at the classical parts of vision: fast, well-defined operations on pixels. It does not replace deep learning frameworks such as PyTorch for training large models, but it usually sits next to them. It loads and prepares the images that feed a CNN, YOLO or ViT model, and it draws and post-processes the results. It is also a bridge to tabular data: measurements taken from images, such as size, color and defect counts, become rows in a table.

What it is

OpenCV stands for Open Source Computer Vision Library. It provides thousands of algorithms for working with images and video. It runs on Linux, Windows, macOS, Android and iOS, and it is designed for real-time work.

Analogy: a well-stocked workshop for pictures. There are tools for cutting (crop), scaling (resize), cleaning (blur and denoise), measuring (contours, moments), comparing (template and feature matching) and watching (video capture and tracking). You choose the tools and build a pipeline.

The image as a numeric array

In OpenCV for Python, an image is a NumPy array. A color image has shape height by width by 3, one number per channel per pixel, usually in the range 0 to 255 stored as unsigned 8-bit integers. A grayscale image has shape height by width. This means all of the NumPy tools work on images, and images can be converted to tables or tensors easily.

A common surprise: OpenCV stores color channels in the order blue, green, red (BGR), not red, green, blue (RGB). Many other libraries use RGB, so conversion is needed when you move images between them.

Main modules

Typical jobs

• Preprocessing images before a neural network.
• Visual inspection of manufactured parts.
• Reading barcodes and measuring objects.
• Document scanning, perspective correction and cleanup.
• Motion detection and tracking in video.
• Camera calibration and measuring in real units.
• Augmenting training data with flips, crops and color shifts.
• Simple face, eye and shape detection with classic methods.

Where it came from

OpenCV began as a research initiative at a large chip company around the end of the 1990s, with the goal of advancing computer vision and showing the use of fast processors. The first public release appeared around 2000. Early versions had a C interface. Over the following years the project moved to an open source community model, a C++ interface was added, and Python bindings grew popular because they pair well with NumPy. A major redesign came with the 2.x series, which made the C++ API central, and another with 3.x and 4.x, which introduced a modular structure, hardware acceleration layers and a deep learning module for running trained networks.

Over time the project added support for GPUs and for many platforms, and it became a de facto standard for teaching and for industrial vision. A non-profit foundation now supports the project. The library is distributed under a permissive open source license, which has helped its use in products. Related efforts include extra modules for newer algorithms, kept in a separate contributions repository.

OpenCV's history mirrors that of the field. In the early years, vision was built from hand-designed features: edges, corners, gradients and statistical classifiers. Well-known methods in the library include Canny edge detection, Hough transforms, SIFT-style and ORB-style keypoints, Haar cascades and histogram of oriented gradients detectors. After deep learning took over many tasks around the early 2010s, OpenCV added the dnn module so that trained networks could be run inside the same pipeline, without a heavy training framework.

How it works

OpenCV pipelines are sequences of simple array operations. The following steps describe a typical visual inspection pipeline and explain each concept.

Step 1: capture or load an image

import cv2
img = cv2.imread("example.internal/parts/part_001.png")   # BGR array

For video or cameras, use a capture object that returns frames one by one. Frames arrive at a certain rate, and the pipeline must keep up if it needs to run in real time.

Step 2: convert color space

Different color spaces suit different jobs.

• Grayscale: one channel, good for edges and shapes.
• HSV: hue, saturation, value. Good for picking colors independent of brightness.
• Lab: closer to human color difference.

Analogy: looking at a photo in black and white helps you notice shapes, while a color wheel helps you pick out the red apples.

Step 3: reduce noise

Cameras add noise. Blurring averages nearby pixels.

• Box and Gaussian blur: smooth the image; Gaussian weighs near pixels more.
• Median blur: replaces each pixel by the median of its neighbors; excellent for salt-and-pepper noise.
• Bilateral filter: smooths while preserving edges.

Step 4: find structure

Thresholding turns a gray image into black and white by choosing a cutoff. Otsu's method picks the cutoff from the histogram automatically. Adaptive thresholding uses a different cutoff for each region, which helps with uneven lighting.

Edge detection finds places where brightness changes sharply. The Canny detector smooths the image, computes gradients, thins edges and links strong ones. Think of it as drawing an outline where the picture changes.

Morphology cleans binary images. Erosion shrinks white regions, removing small specks. Dilation grows them, closing small gaps. Opening (erode then dilate) removes noise. Closing (dilate then erode) fills holes.

Contours are the outlines of connected white regions. Once you have contours you can measure area, perimeter, bounding box, orientation and shape.

Step 5: geometry

Resize, rotate, crop and warp images. A perspective transform maps a skewed rectangle, such as a photo of a page taken at an angle, into a straight rectangle. Camera calibration estimates lens distortion and focal length from pictures of a known pattern, so that pixel distances can be converted to millimeters.

Step 6: features and matching

Keypoints are distinctive spots such as corners. Descriptors summarize the area around a keypoint as a vector. Matching descriptors between two images lets you align them, build panoramas, recognize objects or track movement. Fast binary descriptors such as ORB are free to use and quick. The matching step is a nearest neighbor search, closely related to ideas in FAISS and HNSW.

Step 7: motion

Background subtraction models the static scene and flags pixels that change. Optical flow estimates how pixels move between frames. Trackers follow an object across frames once it has been located.

Step 8: run a neural network

The dnn module loads trained models, for example in ONNX format (see ONNX Runtime), and runs them on CPU or supported accelerators. A typical flow: resize the image, scale pixel values, build a blob, run a forward pass, and read the output. Detection outputs are lists of boxes, classes and confidence scores. Non-maximum suppression removes overlapping boxes.

Step 9: measure and decide

Turn pixel results into numbers: area in square millimeters, count of defects, mean color inside a region. Compare them with thresholds or feed them to a classifier. Then draw results on the image, save it and report.

Step 10: deploy

Pipelines run on edge devices next to cameras or on servers. Considerations include frame rate, latency, lighting control, camera triggers, logging and model updates.

Why classical methods still matter

Neural networks need training data and compute. A defect that is simply a dark scratch on a pale surface may be found by a threshold and a contour filter in a millisecond, with no training, and with results that are easy to explain and audit. Classical steps also clean inputs for neural models and measure their outputs. In practice, the best systems mix both.

Key concepts and terms

A worked example

This example shows a small visual inspection pipeline: find round parts in an image, measure them and flag those outside the size range. The image path is a placeholder.

import cv2
import numpy as np
import pandas as pd

img = cv2.imread("example.internal/parts/batch_17.png")
assert img is not None, "image not found"

gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
blur = cv2.GaussianBlur(gray, (5, 5), 0)

# Otsu threshold finds the cutoff automatically
_, mask = cv2.threshold(blur, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)

# Clean small specks, fill small holes
kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5))
mask = cv2.morphologyEx(mask, cv2.MORPH_OPEN, kernel, iterations=1)
mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, kernel, iterations=2)

contours, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

MM_PER_PIXEL = 0.05          # from calibration
MIN_D, MAX_D = 9.5, 10.5     # allowed diameter range in mm

rows = []
for i, c in enumerate(contours):
    area_px = cv2.contourArea(c)
    if area_px < 200:         # ignore tiny noise
        continue
    perim = cv2.arcLength(c, True)
    circularity = 4 * np.pi * area_px / (perim * perim + 1e-9)
    (x, y), r = cv2.minEnclosingCircle(c)
    diameter_mm = 2 * r * MM_PER_PIXEL
    ok = (MIN_D <= diameter_mm <= MAX_D) and circularity > 0.85
    rows.append({
        "part_id": i, "cx": x, "cy": y,
        "diameter_mm": diameter_mm, "circularity": circularity,
        "mean_gray": float(cv2.mean(gray, mask=cv2.drawContours(
            np.zeros_like(gray), [c], -1, 255, -1))[0]),
        "pass": ok,
    })
    color = (0, 200, 0) if ok else (0, 0, 255)
    cv2.circle(img, (int(x), int(y)), int(r), color, 2)

table = pd.DataFrame(rows)
print(table.describe())
cv2.imwrite("batch_17_checked.png", img)
table.to_csv("batch_17_measurements.csv", index=False)

How the pipeline works:

1. Convert to gray and blur to remove noise.
2. Threshold with Otsu's method to separate parts from the background.
3. Clean the mask with opening and closing.
4. Find contours, one per part.
5. Compute area, perimeter, circularity and enclosing circle.
6. Convert pixels to millimeters using the calibration factor.
7. Mark each part as pass or fail and draw the result.
8. Save the measurements as a table.

The final step is the link to tabular models: the image has become a table of rows and numeric columns, one row per part.

A numeric walk-through of thresholding and Otsu

Take a tiny gray image with pixel values 20, 22, 25, 200, 205, 210. The histogram has two groups. Otsu's method tries every possible cutoff and picks the one that minimizes the variance inside the two groups, which is equivalent to maximizing the variance between them. A cutoff near 100 separates the dark group from the bright group perfectly. Pixels above the cutoff become 255, and the others 0.

A numeric walk-through of a convolution

A 3 by 3 box blur uses a kernel of nine values of 1/9. For a pixel whose 3 by 3 neighborhood has values 10, 20, 30 in each of three rows, the sum is 180 and the blurred value is 20. A Sobel kernel for horizontal gradients uses weights such as -1, 0, 1 across columns. On a patch where the left column is 10 and the right column is 90 in all rows, the response is large, which marks a vertical edge.

Running a detection network

import cv2
import numpy as np

net = cv2.dnn.readNetFromONNX("example.internal/models/detector.onnx")
img = cv2.imread("example.internal/frames/frame_0001.png")

blob = cv2.dnn.blobFromImage(img, scalefactor=1/255.0, size=(640, 640),
                             swapRB=True, crop=False)
net.setInput(blob)
out = net.forward()
print("raw output shape:", out.shape)
# Decode boxes, filter by confidence, then run cv2.dnn.NMSBoxes

The exact decoding depends on the model. Check the documentation of the model you export, such as a YOLO or DETR variant.

Measuring with a calibrated camera

To measure in real units, place an object of known size, such as a ruler or a calibration grid, in the scene and compute millimeters per pixel. For accuracy, correct lens distortion first using the camera matrix and distortion coefficients obtained from calibration. Keep the camera and lighting fixed, since a change in height or angle changes the scale.

Strengths and limitations

Strengths

• Mature, widely documented and stable.
• Fast, with optimized C++ and hardware acceleration.
• Runs on desktops, servers and edge devices.
• Covers a very wide range of classical vision operations.
• Python bindings work smoothly with NumPy, pandas and PyTorch.
• Runs deep models without a full training framework.
• Permissive license suits commercial products.
• Large community and many tutorials.

Limitations

• It is not a training framework for modern deep learning.
• Hand-tuned pipelines are brittle. A change in lighting can break a threshold.
• The API is large and some functions have unintuitive defaults, such as BGR order.
• Documentation quality varies across modules.
• Classical detectors are weaker than modern neural networks on complex scenes.
• GPU support depends on how the library is built.
• Some advanced or patented algorithms may live in separate modules or require care in licensing.
• Real-time performance needs careful engineering of the whole pipeline.

How it compares with alternatives

A practical split of work:

• Use OpenCV for I/O, resizing, color conversion, geometric correction, simple segmentation, measurement and drawing.
• Use deep learning models for recognition, detection and segmentation in variable scenes.
• Use a tabular model on the measurements afterwards for decisions such as pass or fail, grade or risk.

Role in tabular foundation models

OpenCV is not a tabular tool. Its role is to turn pictures into tables, so that models that learn from raw tables can use visual information. In many industrial and commercial settings the images are only one source, and the decision depends on a table that joins image measurements with sensor readings, order data and process settings.

From pixels to rows and columns

A pipeline like the worked example produces one row per object or per image, with numeric columns such as area, diameter, circularity, mean color and defect count, and categorical columns such as camera id, line, shift and defect type. These rows go into the same tables as other process data. Tabular foundation models such as TabPFN, TabICL, TabSTAR, FT-Transformer and SAINT can then predict outcomes from the combined table. See /model/architecture for how such models read mixed columns.

Mixed types and no column names

Image-derived tables mix numbers, categories and flags. Feature names often come from engineers' scripts and differ between lines. A tabular model that learns from values and relationships, without needing column names, is a good fit when many cameras and scripts produce slightly different outputs.

Missing values

Images fail in practice. A part may be out of frame, glare may hide a region, or a frame may be dropped, which leaves missing cells. Missing values should be recorded, not silently replaced. Pre-trained tabular models that expect missing cells, as in Masked cell modeling, and learned imputers from Imputation networks can handle these gaps better than a constant fill. It is wise to keep a flag column that says the image-based feature was missing.

Pre-training on synthetic or real tables

Tabular foundation models are pre-trained on large collections of synthetic or real tables. Image-measurement tables are small and noisy, typical of the few-hundred-row problems where such models are most helpful. A quality team that measures a few thousand parts can get a strong first model without much tuning. For research context, see /model/research, and for evaluation practice, see /model/benchmarks.

In-context learning

In-context learning lets a model use labeled example rows as a prompt. For visual inspection, a handful of recent labeled parts from a new product or a new camera can serve as context, which allows adaptation without a retraining cycle. This is useful during changeovers when the new product has few examples.

Anomaly detection

Many inspection problems have few defect examples. A common design is to learn what normal measurements look like and flag deviations. Anomaly detection with Isolation forest works on the measurement table, and learned densities can score each row. Image-level anomaly methods use neural features, but a table of simple measurements is easier to explain to a quality engineer.

Continual learning

Lighting, wear, suppliers and products change. A model trained last quarter may drift. Continual learning methods such as Experience replay and EWC let a model adapt to new conditions while keeping what it knows. In an OpenCV pipeline, also monitor the inputs: track mean brightness and sharpness, because many drift problems start in the camera, not in the model.

Sector and semantic classification

Vision pipelines in retail, logistics and documents also produce text: labels read by OCR, product names and category tags. Semantic classification turns such text into standard categories or sectors, which become columns for the tabular model. See Semantic sector heads.

Imputation and sensor fusion

A table that joins image measurements to machine sensor readings from protocols such as OPC UA and MQTT is a typical smart manufacturing dataset; see Smart manufacturing and Predictive maintenance. When one source is missing, imputation across sources helps keep the row usable.

Where the vision side stays separate

For recognizing what is in a photo, segmenting objects or reading scenes, vision models do the work. A tabular foundation model does not look at pixels. The clean split is: vision tools extract facts from images, and tabular models reason over the table of facts together with everything else.

Practical guidance and best practices

Control the capture

• Fix the camera, lens, distance and angle.
• Control lighting. Consistent light does more for accuracy than clever code.
• Use diffuse or backlit setups to make edges clean.
• Use a trigger tied to the part position instead of free-running capture when possible.
• Store a few reference images and a calibration target.

Write robust pipelines

• Convert color spaces on purpose, and remember the BGR order.
• Validate inputs: check that the image loaded and has the expected shape.
• Avoid magic numbers. Keep thresholds and sizes in a configuration file.
• Prefer adaptive or Otsu thresholds over fixed ones when lighting varies.
• Work in a region of interest to save time and avoid distractions.
• Keep intermediate images for debugging, and save failures for review.

Measure correctly

• Calibrate and correct distortion.
• Convert pixels to real units with a known reference.
• Report measurement uncertainty by repeating measurements on a reference part.
• Avoid measuring at the image edge where distortion is strongest.

Performance

• Resize early if full resolution is not needed.
• Use vectorized NumPy and OpenCV calls instead of Python loops over pixels.
• Use multi-threading or hardware acceleration where available.
• Profile the pipeline. Capture and decoding often dominate.

Combine with learning

• Use OpenCV augmentation or a library built on it to enrich training sets.
• Match preprocessing between training and inference exactly: same resize method, same color order, same scaling.
• Export the measurement table with image ids so every row can be traced back to its picture.
• Keep a human review loop for borderline cases and use the reviewed labels for retraining.

Deployment

• Pin the library version and build options.
• Log processing time, failures and input statistics.
• Test with images from every shift and season.
• Plan for secure handling of images that may contain people or sensitive information, and follow applicable privacy rules.

A short end-to-end checklist

1. Define what must be measured or decided, and the tolerance.
2. Fix camera, lens, mounting and lighting before writing code.
3. Capture a representative image set from all shifts, products and conditions.
4. Calibrate and record the scale in real units.
5. Build the pipeline step by step, saving intermediate images.
6. Test on the whole image set, not on a few favorites.
7. Output a table with one row per object and an image id.
8. Compare results with manual measurements on a reference set.
9. Add monitoring for brightness, sharpness and failure rates.
10. Plan a review loop for borderline cases and for retraining.

Typical processing budget

For a real-time line, plan the time per frame. As a rough guide, capture and decoding, preprocessing, the main algorithm, post-processing and communication each take a share of the budget. If the line produces ten parts per second, the whole path must finish in well under a hundred milliseconds per part. Measure each stage and optimize the slowest first. Reducing image size, restricting to a region of interest and avoiding copies of large arrays usually give the biggest wins.

Choosing between thresholds, features and networks

Common mistakes

1. Forgetting BGR order. Colors look wrong, or a model trained on RGB performs poorly.
2. Reading a missing file without checking. The result is None, and errors appear later.
3. Fixed thresholds with changing light. Works in the lab, fails on the line.
4. Using the wrong data type. Operations on unsigned 8-bit arrays can overflow or clip.
5. Resizing with a different method in training and inference. Small differences change model outputs.
6. Skipping calibration. Measurements in pixels are not measurements in millimeters.
7. Measuring with a tilted camera. Perspective distortion gives inconsistent sizes.
8. Looping over pixels in Python. Very slow compared with vectorized calls.
9. Not saving failure cases. You cannot improve what you cannot inspect.
10. Mixing up coordinates. OpenCV points are (x, y), while array indices are (row, column).
11. Ignoring compression. Heavy JPEG compression adds artifacts that confuse edges.
12. Treating the pipeline as finished. Cameras age, lenses get dirty and products change.

Related video and tracking tasks

Video adds time. A few common patterns are worth knowing.

• Frame differencing: subtract consecutive frames to see what moved. Simple and fast, but sensitive to lighting flicker.
• Background subtraction: learn a model of the empty scene and mark pixels that differ. Works for fixed cameras, such as a conveyor or a doorway.
• Tracking by detection: detect objects in every frame and link detections across frames by overlap or appearance.
• Counting: define a line or zone and count objects that cross it. Useful for logistics and traffic.
• Dwell time: measure how long an object stays in a zone, which becomes a numeric column for later analysis.

Each of these ends in numbers per object or per time window, so the output again fits a table: timestamp, camera id, zone, count, speed, dwell time.

Frequently asked questions

What is OpenCV used for?

OpenCV is used to process and analyze images and video. Common uses are preprocessing for neural networks, visual inspection in factories, measuring objects, reading documents, tracking motion and calibrating cameras. It provides fast, well-tested building blocks for these tasks.

Is OpenCV a machine learning library?

Not mainly. It is a computer vision library with classical image processing at its core. It includes a small machine learning module and a deep learning inference module that runs trained models, but training large models is usually done in frameworks such as PyTorch or TensorFlow.

Why are OpenCV colors in BGR order?

For historical reasons, OpenCV stores color channels as blue, green, red. Most other libraries use red, green, blue. When you pass an image between OpenCV and another tool, convert the order with a color conversion call, or colors and model inputs will be wrong.

Can OpenCV run deep learning models?

Yes. The dnn module loads trained networks from several formats, including ONNX, and runs them for inference on CPU and supported accelerators. It is convenient for deployment, though it covers fewer operators and optimizations than dedicated runtimes, so check that your model is supported.

Should I use OpenCV or a deep learning model for inspection?

Use the simplest tool that is reliable. If a defect is easy to define with thresholds, edges or shapes and lighting is controlled, OpenCV alone can be fast and explainable. If defects vary a lot or the scene is cluttered, a trained detector or segmentation model is better. Many systems combine them.

How does OpenCV relate to tabular data?

OpenCV turns images into measurements such as size, shape, color and counts, which can be stored as rows in a table. Those rows can be joined with sensor, order and process data and passed to a tabular model for prediction or anomaly detection. Keep an image id on each row to trace back to the picture.

Can tabular foundation models read images?

No. They work on rows and columns. They need image information turned into numbers first, either by classical measurements from OpenCV or by embeddings from a vision model. The image features then become columns like any other.

How do I measure real sizes with OpenCV?

Calibrate the camera with a known pattern, correct lens distortion, fix the camera position and compute a scale in millimeters per pixel using an object of known size. Then convert pixel measurements with that scale. Repeat measurements on a reference part to estimate uncertainty.

Is OpenCV free for commercial use?

OpenCV is released under a permissive open source license that allows commercial use, but you should read the license terms for the version you use. Some optional contributed modules may have different terms, so review them before shipping a product.

What is the best way to learn OpenCV?

Start with reading, resizing, color conversion, blurring and thresholding on your own images. Then try edges, contours and morphology, and build a small measurement pipeline like the example here. After that, add video, feature matching and running a pretrained network.

Glossary

• Adaptive threshold: A threshold computed locally for each region.
• BGR: Blue, green, red channel order used by OpenCV.
• Canny: A popular multi-stage edge detector.
• Contour: The outline of a connected region.
• Convolution: Applying a kernel across an image.
• Dilation: Growing white regions in a binary image.
• Erosion: Shrinking white regions in a binary image.
• Gaussian blur: Smoothing with a bell-shaped weighting.
• Histogram: A count of pixels per intensity.
• Homography: A mapping between two views of a plane.
• Keypoint: A distinctive point in an image.
• Morphology: Shape-based operations on binary images.
• Non-maximum suppression: Removing overlapping duplicate detections.
• Optical flow: Estimated pixel motion between frames.
• Otsu's method: An automatic way to choose a threshold.
• ROI: Region of interest.
• Thresholding: Converting a gray image to black and white by a cutoff.

Summary

OpenCV is a mature, fast, open source library for classical computer vision and for the plumbing around modern vision models. It reads and prepares images, filters and segments them, measures shapes, matches features, tracks motion, calibrates cameras and runs trained networks. It shines when lighting and capture are controlled, when speed and explainability matter, and when a pipeline must run close to the camera.

For tabular foundation models, OpenCV is a feeder. It converts images into tables of measurements that can be joined with sensor and process data, imputed, checked for anomalies and used for prediction, with in-context learning handling new products and continual learning handling drift. Control the capture, calibrate, keep preprocessing identical between training and inference, record missing values explicitly and trace every row to its image. For more background see /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Vision and multimodal in my stack, under Foundation models and applied ML. Others in the same group: