CLIP

Overview

CLIP, short for Contrastive Language-Image Pre-training, is a model that learns to place images and text in the same embedding space. Pictures and their captions land close together, and mismatched pairs land far apart. Once trained, you can classify an image without any task-specific training by comparing its embedding with the embeddings of text descriptions of the possible classes. You can also search images with a sentence, or search text with an image.

CLIP made zero-shot image classification practical and became a core building block for image search, dataset labeling, content filtering and many multimodal systems. It does not draw boxes or masks, and it does not generate text. For localization pair it with YOLO, DETR or SAM. This page explains the idea, the history, the mechanics, a worked example, and how the joint-embedding idea relates to models that learn from raw tables.

What it is

CLIP is a pair of encoders trained together: one for images and one for text. Each turns its input into a vector of the same length. The training goal is simple: vectors for matching image and caption pairs should be similar, and vectors for non-matching pairs should be different.

Components

• An image encoder, either a convolutional network or a vision transformer, see ViT.
• A text encoder, a transformer that reads a short piece of text.
• A projection step that maps both outputs into a shared space.
• A learned temperature that scales similarity scores.

What it outputs

CLIP outputs embeddings, not labels. A label appears only when you compare an image embedding with text embeddings of candidate labels and pick the closest.

Everyday analogy

Imagine a huge library where every photo has a caption stuck to it. You shuffle the photos and captions on a table, then train a helper to pair them up. After millions of rounds, the helper understands what kinds of pictures go with what kinds of words. Now you can hand the helper a new photo and a list of possible captions, and the helper picks the best match, even for captions it has never practiced with.

Where it came from

Before CLIP

Image classifiers were trained on fixed label sets with human-annotated categories. They were accurate on those categories and weak elsewhere. Earlier work on learning from images with text, such as tags and captions, existed, but at smaller scale or with different training objectives.

The CLIP paper

CLIP was introduced in 2021 in the paper "Learning Transferable Visual Models From Natural Language Supervision". It trained on a very large set of image and text pairs collected from the internet, with a contrastive objective. The key finding was that natural language provides a flexible supervision signal: instead of a fixed list of classes, the model learns from the open-ended descriptions people write.

Why it was a turning point

• Zero-shot transfer: the model matched or approached supervised baselines on many datasets without using their training labels.
• Robustness: it was often more stable under distribution shifts than models trained on a single labeled dataset.
• Flexibility: new classes only need new text.

Follow-ups and open reproductions

Open source reproductions trained on public image and text datasets appeared, often under the name OpenCLIP, with larger models and different data. Variants changed the loss, for example using a sigmoid on each pair instead of a softmax across the batch, which scales more easily. Others added captioning heads, better data filtering, or finer-grained alignment. CLIP-style encoders also became the visual front end for many image-to-text and text-to-image systems, including BLIP-2, LLaVA and Stable Diffusion style generators. Check current papers for exact versions and numbers.

How it works

Step 1: Collect image and text pairs

Training data is a large collection of images, each with a piece of text such as a caption or alt text. The text is noisy and varied. That is a feature, not a bug: the diversity teaches the model many concepts.

Step 2: Encode each side

Each image goes through the image encoder to produce a vector. Each text goes through the text encoder to produce a vector. Vectors are normalized to unit length, so comparing them means measuring the angle between them. Cosine similarity is simply the dot product of unit vectors.

Step 3: Build a batch similarity matrix

Take a batch of N image and text pairs. Compute the similarity of every image with every text in the batch. This gives an N by N table. The diagonal holds the true pairs. Off-diagonal entries are mismatched pairs.

Analogy: a seating chart where each guest should find their own name card. The diagonal is the correct seat. Every other seat is wrong.

Step 4: Contrastive loss

For each image, the model should pick its own caption out of the N captions in the batch. For each caption, the model should pick its own image out of the N images. This is a cross entropy over each row and each column of the similarity matrix. The sum of the two is the symmetric loss. It is a form of the InfoNCE objective, see InfoNCE.

The result: matching pairs are pulled together, and all other pairs in the batch are pushed apart.

Reading the similarity matrix

Look at a toy batch of three pairs. Rows are images, columns are captions, and each cell is a cosine similarity.

The diagonal is high and the rest is low, which is what training aims for. The loss looks at each row and asks "did this image pick its own caption", then at each column and asks "did this caption pick its own image". A model that makes the diagonal stand out gets a low loss.

Hard negatives

Easy negatives, such as a photo of a dog against a caption about a spaceship, teach little. Hard negatives, such as two similar dog breeds, teach a lot. Large batches contain more hard negatives by chance. Some training recipes also mine hard negatives on purpose. For your own fine-tuning, include look-alike classes in the same batch.

Image resolution and crops

CLIP-style models take a fixed input size. A wide photo is resized or center-cropped, which can cut off the object of interest. For large images with small details, tile the image into overlapping crops, embed each, and combine the scores. Taking the maximum over tiles finds a small defect. Averaging over tiles describes the whole scene.

Text length

The text encoder has a limit on how many tokens it reads, so a long description is truncated. Keep prompts short and put the most important words first. For long documents, split the text into sentences and embed each one.

Dimension and storage

Embeddings are vectors of a few hundred to about a thousand numbers. A million images at 512 numbers each in 32-bit floats take about two gigabytes. Reducing to half precision or using product quantization cuts that further, at a small cost in accuracy. Plan storage and index memory early.

Step 5: Temperature

Similarity scores are divided by a temperature before the softmax. A small temperature sharpens the differences and makes the model care more about the hardest negatives. In CLIP the temperature is a learned parameter, not a fixed one.

Step 6: Large batches matter

Each batch supplies N minus 1 negatives for every pair. A bigger batch means more and harder negatives, which helps learning. This is why CLIP-style training uses very large batches across many devices. See data parallelism for the idea of splitting a batch across devices.

Step 7: Zero-shot classification

To classify an image into K classes:

• Write a text prompt for each class, for example "a photo of a scratch" or "a photo of a dent".
• Encode all prompts with the text encoder, once.
• Encode the image with the image encoder.
• Compute cosine similarity between the image vector and each prompt vector.
• Apply a softmax to get probabilities, and pick the highest.

No training on your classes is needed. Prompt wording matters, and averaging several phrasing variants often helps.

Step 8: Retrieval

For image search, encode all images once and store their vectors in an index. Encode the search sentence and find the nearest image vectors. This works with a vector database or an approximate nearest neighbor method such as HNSW, and it is a standard recipe for cross-modal search. The reverse also works: search captions with an image.

Step 9: Linear probes and fine-tuning

If you do have labels, you can train a small classifier on frozen image embeddings, called a linear probe. This often beats zero-shot on your domain with little data. Full fine-tuning can do better but risks losing the general alignment, so many teams use adapters or low learning rates, see adapters.

Step 10: Use as a building block

The image embeddings or the text embeddings are fed into other systems:

• A language model that can answer questions about an image, see LLaVA and BLIP-2.
• A generator that is guided by a text embedding, see Stable Diffusion and ControlNet.
• A scoring function that ranks generated or retrieved images against a prompt.

What the shared space gives you

Because both modalities live in one space, simple vector math has meaning. The distance between an image and a sentence is a relevance score. The average of several image vectors is a prototype for a class. A direction in the space can approximate a concept. These properties are approximate and depend on the data, but they make CLIP unusually flexible.

Why zero-shot works at all

Training on open-ended text taught the model many concepts and the words for them. At test time a class name is just a short text. If the model has seen enough pictures with that word or similar words, the text vector lands near the matching images. Rare, technical or domain-specific concepts, such as industrial defect types, may be poorly covered, which is why domain testing is essential.

Key concepts and terms

A worked example

Goal: build a quick zero-shot filter that labels product photos as "damaged", "intact" or "unclear", then compare with a few human labels before deciding whether to train a dedicated model.

Step 1: Write prompts

Use several phrasings per class:

• damaged: "a photo of a damaged package", "a photo of a dented box"
• intact: "a photo of an intact package", "a photo of an undamaged box"
• unclear: "a blurry photo", "a photo with nothing visible"

Step 2: Code sketch

The sketch follows the common Hugging Face transformers pattern. Names are placeholders.

import torch
from transformers import CLIPModel, CLIPProcessor

model = CLIPModel.from_pretrained("clip-base-placeholder")
proc = CLIPProcessor.from_pretrained("clip-base-placeholder")
model.eval()

prompts = {
    "damaged": ["a photo of a damaged package", "a photo of a dented box"],
    "intact":  ["a photo of an intact package", "a photo of an undamaged box"],
    "unclear": ["a blurry photo", "a photo with nothing visible"],
}

# text side: encode once, average prompts per class
class_vecs = []
names = []
with torch.no_grad():
    for name, texts in prompts.items():
        t = proc(text=texts, return_tensors="pt", padding=True)
        v = model.get_text_features(**t)
        v = v / v.norm(dim=-1, keepdim=True)
        class_vecs.append(v.mean(dim=0))
        names.append(name)
class_mat = torch.stack(class_vecs)
class_mat = class_mat / class_mat.norm(dim=-1, keepdim=True)

Step 3: Classify an image

from PIL import Image

img = Image.open("photo_001.jpg")
with torch.no_grad():
    px = proc(images=img, return_tensors="pt")
    iv = model.get_image_features(**px)
    iv = iv / iv.norm(dim=-1, keepdim=True)
    logits = 100.0 * iv @ class_mat.T     # scale like the learned temperature
    probs = logits.softmax(dim=-1)[0]

for n, p in zip(names, probs.tolist()):
    print(n, round(p, 3))

Step 4: A numeric walk-through

Suppose the cosine similarities between one image and the three class vectors are 0.31, 0.27 and 0.20. With a scale of 100, the logits are 31, 27 and 20. A softmax gives roughly:

• damaged: e to the 31 divided by the sum. Compared with the second class the gap is 4, so the first is about 55 times larger than the second.
• The first class gets about 98 percent of the probability, the second about 2 percent, and the third almost zero.

A small difference in cosine similarity becomes a confident probability after scaling. That is why CLIP probabilities can look overconfident and should be calibrated before being used as real probabilities.

Step 5: Evaluate against human labels

Label 200 photos by hand. Compute accuracy and confusion matrix per class. If "unclear" is poorly handled, add a rule such as "if the top probability is below 0.6, send to a person".

Step 6: Improve without heavy training

• Add more prompt variants that reflect your real photos.
• Include context words like "on a warehouse shelf".
• Train a linear probe on frozen image embeddings using your 200 labels.
• Compare the linear probe with the zero-shot result.

Step 7: Index embeddings for search

Store each image embedding. A staff member can then type "crushed corner" and see the nearest photos. The index can live in a vector store, see Milvus and HNSW for common approaches to nearest-neighbor search.

Step 8: Put outputs in a table

Store per image: id, top label, probabilities, embedding reference, model version, prompt set version and timestamp. Roll up per shipment. This table feeds downstream models that predict claims or returns.

Strengths and limitations

Strengths

• Zero-shot classification with only text descriptions.
• Flexible: new classes need no retraining.
• One model supports classification, retrieval and ranking.
• Good generic features for many downstream tasks via linear probes.
• Often robust under distribution shifts compared with narrow supervised models.
• Embeddings are small and easy to index and cache.
• Serves as the visual or text front end for many multimodal systems.

Limitations

• No localization. It scores whole images, not regions.
• Fine-grained and technical categories can be weak, such as specific defect types or species.
• Counting, spatial relations and negation are unreliable. "A cup to the left of a book" and the reverse can score similarly.
• Sensitive to prompt wording.
• Reflects biases in internet-scale training data, which can lead to unfair or stereotyped associations.
• Probabilities can be overconfident and need calibration.
• Reading text inside images is limited compared with dedicated OCR.
• Domain shift, such as medical or satellite images, often needs adaptation.
• Training from scratch needs very large data and compute.

Honest remarks on benchmarks

Reported zero-shot accuracy on public datasets does not carry over to your images. Measure on a labeled sample from your domain, and report per-class results, not only the average.

How it compares with alternatives

Combining with localization tools

CLIP tells you what a crop looks like. A detector or segmenter tells you where. A common pipeline: SAM proposes masks, CLIP scores each cropped region against text, and the best match gets the label. Open-vocabulary detectors use a similar principle, aligning region features with text embeddings.

CLIP versus generative vision-language models

Generative models such as LLaVA and BLIP-2 produce free-form text and can explain what they see. CLIP only scores similarity. CLIP is cheaper, faster and well suited for large-scale filtering and retrieval. A generative model is better for reasoning and descriptions. Many systems use CLIP-like encoders inside generative models.

Choosing quickly

• Many images, a changing label list, no training data: CLIP zero-shot.
• Fixed label list with plenty of labeled data: a trained classifier, perhaps starting from CLIP features.
• Need boxes or masks: detector or segmenter, optionally with CLIP for labels.
• Need explanations in words: a generative vision-language model.

Role in tabular foundation models

CLIP is a vision-language model, not a table model. Still, its ideas appear in table-native work in several concrete ways. The pages on research, architecture and benchmarks describe how a model that learns from raw tables is built and tested.

Contrastive alignment and the same loss

The training objective in CLIP is InfoNCE, a contrastive loss. The same family of losses is used to learn useful row or cell representations in tables, for example by treating two views of a row, one with some cells masked or noised, as a positive pair and other rows as negatives. See InfoNCE and denoising autoencoders. The intuition carries over directly: pull matching views together, push others apart.

Text meets tables

Real tables often contain text columns: product titles, job descriptions, free-form notes, and column names. Models like TabSTAR bring language understanding into tabular learning, so that the meaning of words in cells and headers helps prediction. CLIP shows the value of a shared semantic space between modalities. A text-tabular model uses a similar idea: map text and numbers into a space where related items are close.

Zero-shot labeling for semantic and sector classification

CLIP classifies by comparing an item with text descriptions of classes. Table-native systems with semantic sector heads do something related: they map rows or entities to a taxonomy of sectors or categories using semantic similarity between the row and label descriptions. Because label descriptions are text, new labels can be added without retraining from scratch, which mirrors CLIP's flexibility.

Embeddings as table columns

CLIP image embeddings can be reduced to a few numeric columns, for example by clustering or projection with UMAP, and joined with other columns such as price, date or supplier. A tabular model can then use visual information along with structured data. Alternatively, store a similarity score to a text prompt as a single numeric column, such as "similarity to damaged package". It is an interpretable, compact feature.

Retrieval and in-context examples

Tabular foundation models such as TabPFN and TabICL predict from labeled example rows given as context. When the table is huge, you can pick the most relevant context rows by nearest-neighbor retrieval in an embedding space. This is the same retrieval pattern CLIP enables for images. It lets the model see only the most similar examples, within its context limit.

Missing values and imputation

A product table may lack a category but have an image. A CLIP-based zero-shot label can fill or suggest that cell. Learned imputation methods, see masked cell modeling and imputation networks, can then use the suggested value as an input, along with a flag that it was inferred.

Anomaly detection

Embeddings make anomaly detection easy to frame: items far from the cluster of normal ones are suspicious. Run isolation forest, one-class SVM or clustering on embeddings, or on tabular summaries of them. See also the general page on anomaly detection.

Pre-training on synthetic data

CLIP was pre-trained on a huge, diverse set of real pairs. Many tabular foundation models are pre-trained on large numbers of generated tables instead, because real tables are often private or scarce. See synthetic data generation. In both cases the aim is breadth: a model that has seen many kinds of structure adapts quickly to a new one.

Continual learning

Updating a CLIP-style model on a new domain can erode its general alignment. The same forgetting problem affects tabular models adapted to new data. Elastic weight consolidation, learning without forgetting and experience replay are standard ways to limit it, and adapters avoid touching the base weights.

Where the analogy stops

CLIP pairs two modalities with natural one-to-one correspondence between an image and its caption. A table has no such built-in pairing, and rows are usually independent examples. The transferable ideas are the shared semantic space, contrastive pre-training and zero-shot use through descriptions.

Practical guidance and best practices

Prompts

• Use a template such as "a photo of a {class}" and compare variants.
• Include the context of your domain: "a close-up photo of a scratch on metal".
• Ensemble several prompts and average their embeddings.
• Add a catch-all class like "something else" to reduce forced wrong choices.
• Keep a versioned list of prompts with results.

Evaluation

• Build a labeled sample from your real data.
• Report per-class precision and recall and the confusion matrix.
• Calibrate probabilities, for example with temperature scaling on a validation set.
• Use abstention: send low-confidence items to review.

Adaptation

• Start with zero-shot, then a linear probe on frozen embeddings.
• Fine-tune only if needed, with a small learning rate, and test for loss of general ability.
• Prefer adapters for domain shifts.
• Use active learning to pick which items to label next.

Scaling retrieval

• Normalize embeddings and use cosine or inner-product search.
• Use an approximate nearest neighbor index, see HNSW and hybrid search for combining vector and keyword matching.
• Store the model version with every embedding. Embeddings from different models or versions are not comparable.
• Re-embed the whole collection when you change models.

Serving

• Encode text prompts once and cache them.
• Batch image encoding on a GPU, with mixed precision.
• Measure end-to-end latency, including image decoding and resizing.

Debugging a weak zero-shot result

• Print the top three classes for several failures and see if the confusion is sensible.
• Rewrite prompts using the words your staff actually use.
• Check the image preprocessing matches the model's expected size and normalization.
• Look at whether the target is tiny in the frame, and try tiling.
• Try a larger encoder if latency allows.
• If all else fails, label a few hundred images and train a linear probe.

Fairness and safety

• Test for biased behavior on groups relevant to your use.
• Avoid using zero-shot labels for decisions about people without strong validation.
• Document limits and keep a human review path.

Example of an embedding-score table

The margin is the gap between the top two similarities. A small margin means the choice is shaky, so the item goes to review. A row with missing scores means the image failed to load, and it also goes to review. Columns like these are easy for a tabular model to use and easy for a person to audit.

A short checklist for a first project

• Collect a labeled sample from your own data.
• Write and test at least three prompt variants per class.
• Add a catch-all class and a review rule.
• Compare zero-shot with a linear probe.
• Calibrate scores and pick thresholds by cost.
• Version prompts, models and embeddings.
• Plan re-embedding when models change.

Working with the table side

• Store scores and embeddings with clear versions.
• Use a stable schema and mark missing images explicitly.
• Explain downstream model decisions with attribution tools such as SHAP.

Common mistakes

• Assuming zero-shot accuracy from a public benchmark will hold on your images.
• Using a single prompt and a single wording without testing alternatives.
• Treating CLIP probabilities as calibrated probabilities.
• Mixing embeddings from different model versions in one index.
• Expecting CLIP to localize objects or count them.
• Ignoring that class names can be ambiguous, for example "crane" the bird or the machine.
• Fine-tuning heavily on a small set and destroying the general alignment.
• Forgetting to normalize embeddings before computing similarity.
• Using it for sensitive decisions about people without checking bias.
• Skipping a human review path for low-confidence cases.

Frequently asked questions

What does CLIP stand for?

CLIP stands for Contrastive Language-Image Pre-training. The name describes the training method: a model learns from pairs of images and text by pulling matching pairs together in an embedding space and pushing non-matching pairs apart, which is called contrastive learning.

How does CLIP do zero-shot classification?

You write a short text description for each possible class, embed those descriptions with the text encoder, embed the image with the image encoder, and pick the class whose text embedding is most similar to the image embedding. No training on your classes is required, though prompt wording and domain match affect the result.

Can CLIP generate text or images?

No. CLIP produces embeddings and similarity scores. Its encoders are often used inside systems that generate text or images, such as image-to-text assistants and diffusion models, but CLIP itself only measures how well an image and a text match.

Can CLIP find where an object is in a picture?

Not by itself, because it scores the whole image. To localize, combine it with a detector or a segmenter, then use CLIP to score or label each cropped region. Some open-vocabulary detectors apply the same text-image alignment idea at the region level.

Why does CLIP need such large batches?

The contrastive loss uses the other items in the batch as wrong answers for each pair. Larger batches give more and harder negatives, which improves the learned space. That is why CLIP-style training is distributed over many devices, and why some later variants change the loss to scale more easily.

Is CLIP good for image search?

Yes. Encode every image once, store the vectors in an index, and encode the search text at query time to find the nearest images. It works well for open-ended queries, although fine details, counts and spatial relations can be handled poorly, so evaluate it on your own queries.

Do I need to fine-tune CLIP for my domain?

Not always. Try zero-shot first, then a linear probe on frozen embeddings with a small labeled set. Fine-tune only if those are not enough, using a small learning rate or adapters to avoid losing general knowledge. Always compare against the zero-shot baseline.

Does CLIP have biases?

Yes. It learns from large amounts of internet data, which contain social biases and stereotypes, and those can show up in its associations and rankings. Test it on the groups and cases that matter for your use, and avoid using it unreviewed for decisions that affect people.

How does CLIP relate to tabular foundation models?

Both rely on pre-training for broad skills and on use-time adaptation without new training. CLIP's contrastive loss is also used to learn row and cell representations in tables, its image embeddings can become numeric columns, and its zero-shot labeling by text description is similar to semantic classification of table rows into sectors.

Can I use CLIP to read text in images?

It has some ability to recognize text in images, but it is not a replacement for dedicated optical character recognition. For documents, receipts or labels where exact characters matter, use an OCR system and then use CLIP for the visual content.

Glossary

• Contrastive learning: training by pulling matching pairs together and pushing others apart.
• Cosine similarity: similarity based on the angle between two vectors.
• Cross-modal retrieval: searching one data type using another, such as images with text.
• Embedding: a vector that represents an item's meaning.
• InfoNCE: a contrastive loss that identifies the true pair among negatives.
• Joint embedding space: a single space shared by several modalities.
• Linear probe: a simple classifier trained on frozen features.
• Multimodal: involving more than one data type, such as images and text.
• Negative pair: an image and text that do not match.
• Positive pair: an image and text that do match.
• Prompt ensembling: averaging embeddings of several prompt wordings.
• Temperature: a scale that controls how sharp similarity scores are.
• Zero-shot: performing a task without task-specific training.

Summary

CLIP learns a shared embedding space for images and text by training on image and caption pairs with a contrastive loss. This lets you classify images zero-shot using text descriptions, search images with sentences, and reuse its encoders in larger multimodal systems. It does not localize objects, count reliably or generate text, and it needs domain testing, calibration and bias checks.

Pair it with YOLO, DETR or SAM for where, and with BLIP-2 or LLaVA for explanations. On the tabular side, its contrastive training, shared semantic space and retrieval pattern echo in models such as TabSTAR, TabPFN and TabICL, which learn from raw tables, handle text columns and adapt from examples in context.

Related

It belongs to Vision and multimodal in my stack, under Foundation models and applied ML. Others in the same group: