CLIP: contrastive learning of images and captions in one space
CLIP learned to connect pictures and words from 400 million image–caption pairs. It trains an image encoder and a text encoder together so that matching pairs point the same way and every other pairing in the batch points elsewhere. The result is one embedding space shared by images and text, the basis of zero-shot classification and of text-to-image search.
Level IntermediateAbout 12 minRead first: Embeddings, Loss functions1 challenge in the Lab
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
Each of the N images and N captions is a small feature vector built from a shared hidden meaning, with noise on the caption side. Two linear encoders map them onto a circle, the shared embedding space: circles are images, squares are captions, and a line joins each true pair. The matrix shows every image–caption similarity; training should light up the diagonal. Each step applies the symmetric contrastive loss: every image must pick its caption out of the N captions, and every caption its image.
Things to try
- Press Train: the matching pairs rotate towards each other and the diagonal of the matrix lights up.
- Raise the temperature to 1: the softmax becomes flat. Pairs can still be matched, but the loss stays high because no match is ever confident.
- Set caption noise to 0.8: captions describe their images only loosely and accuracy stalls below 100%.
- Increase the batch to 16: more negatives per step, but the circle gets crowded.
- Press New batch and train again with a learning rate of 1: training is faster but can overshoot.
How it works in real systems
The real CLIP uses a vision transformer or ResNet for images and a transformer for text, 512- to 1,024-dimensional embeddings, a learnable temperature and batches of 32,768 pairs spread over many GPUs. Zero-shot classification embeds a prompt such as 'a photo of a dog' for each class and picks the class whose text is closest to the image. Successors such as SigLIP replace the softmax with a per-pair sigmoid loss, which works better with smaller batches.
Key formulas
$$\mathcal L=-\frac1{2N}\sum_{i=1}^{N}\Big[\log\frac{e^{s_{ii}/\tau}}{\sum_j e^{s_{ij}/\tau}}+\log\frac{e^{s_{ii}/\tau}}{\sum_j e^{s_{ji}/\tau}}\Big],\qquad s_{ij}=\hat u_i^{\top}\hat v_j$$
The symmetric InfoNCE loss of CLIP over a batch of N pairs.
Common mistakes
- Small batches: contrastive learning gets its signal from the negatives in the batch.
- Fixing the temperature by hand: CLIP learns it, starting from 0.07.
- Expecting CLIP embeddings to encode counting or word order well: contrastive captions reward topic matching.
Real-world numbers
- CLIP was trained on 400 million image–text pairs with a batch size of 32,768.
- Zero-shot CLIP matched the original ResNet-50's ImageNet accuracy without using any of its 1.28 million labelled training images.
- CLIP's learnable temperature is clipped so that the logit scale does not exceed 100.
Further reading
- Learning Transferable Visual Models From Natural Language Supervision (CLIP)
- Representation Learning with Contrastive Predictive Coding (InfoNCE)
- Sigmoid Loss for Language Image Pre-Training (SigLIP)
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · The contrastive loss from scratch
Symmetric InfoNCE over a batch, in PyTorch, on synthetic paired features.
pip install torch transformers pillow requests
import torch, torch.nn.functional as F
torch.manual_seed(0)
N, d = 8, 6
z = torch.randn(N, 3) # shared meaning of each pair
images = z @ torch.randn(3, d) + 0.1 * torch.randn(N, d)
captions = z @ torch.randn(3, d) + 0.4 * torch.randn(N, d)
img_enc, txt_enc = torch.nn.Linear(d, 4), torch.nn.Linear(d, 4)
opt = torch.optim.Adam([*img_enc.parameters(), *txt_enc.parameters()], lr=0.05)
tau = 0.1
for step in range(300):
u = F.normalize(img_enc(images), dim=1)
v = F.normalize(txt_enc(captions), dim=1)
logits = u @ v.T / tau
target = torch.arange(N)
loss = (F.cross_entropy(logits, target) + F.cross_entropy(logits.T, target)) / 2
opt.zero_grad(); loss.backward(); opt.step()
acc = (logits.argmax(1) == target).float().mean()
print(f'loss {loss.item():.3f}, image-to-caption accuracy {acc:.0%}')
Step 2 · Zero-shot classification with a pre-trained CLIP
Embed one prompt per class and pick the closest to the image.
from transformers import CLIPModel, CLIPProcessor
from PIL import Image
import requests
model = CLIPModel.from_pretrained('openai/clip-vit-base-patch32')
processor = CLIPProcessor.from_pretrained('openai/clip-vit-base-patch32')
url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)
labels = ['a photo of a cat', 'a photo of a dog', 'a photo of a car']
inputs = processor(text=labels, images=image, return_tensors='pt', padding=True)
probs = model(**inputs).logits_per_image.softmax(dim=1)[0]
for label, p in zip(labels, probs):
print(f'{p:.3f} {label}')
The picture shows two cats, so 'a photo of a cat' should win by a wide margin, without CLIP ever being trained on these labels.