BLIP-2

Overview

BLIP-2 is a vision-language model that lets a language model "see" images without retraining either the image encoder or the language model. It places a small, trainable bridge called the Querying Transformer, or Q-Former, between a frozen image encoder and a frozen language model. Only the bridge is trained, so the cost is a small fraction of training a full multimodal model from scratch.

The practical result is a model that can caption images, answer questions about them and support image-text retrieval. The same idea matters beyond pictures. BLIP-2 shows how to connect a pre-trained perception module to a pre-trained reasoning module with a thin adapter. That pattern is very close to how modern systems connect raw tables, documents and images to large pre-trained models, which is why it appears in discussions of tabular foundation models and multimodal data pipelines.

What it is

BLIP-2 is a pre-training strategy and a model family for vision-language tasks. The name stands for Bootstrapping Language-Image Pre-training, version 2. It is made of three parts.

• A frozen image encoder, which turns pixels into a set of feature vectors.
• A Q-Former, a small transformer with a fixed number of learned query vectors.
• A frozen large language model, which produces text.

The Q-Former is the only part that is trained from scratch in the main recipe. It reads the image features and compresses them into a short sequence of vectors that the language model can understand as if they were a soft prompt.

Everyday analogy

Think of a museum guide who speaks one language and a visitor who speaks another. You do not teach the guide a new language and you do not teach the visitor either. You hire an interpreter. The interpreter learns what the visitor tends to look at and phrases it in a way the guide understands. The Q-Former is that interpreter.

What it can do

• Image captioning, writing a sentence that describes a photo.
• Visual question answering, answering a question about an image.
• Image-text retrieval, finding the image that matches a text or the text that matches an image.
• Instructed zero-shot image-to-text generation, where a text instruction steers the output.

What it is not

• It is not an image generator. It reads images and writes text.
• It is not a single fixed checkpoint. Different versions pair the same bridge with different language models.
• It is not a tabular model. It works on pixels and text, although its design pattern carries over to table inputs.

Where it came from

BLIP-2 was introduced in 2023 as a follow-up to the earlier BLIP model from 2022. Both came from the line of work on joint image and text pre-training that also includes CLIP, ALIGN and later large multimodal assistants.

The problem it addressed

By 2022, vision-language models were getting expensive. Training end-to-end meant updating billions of parameters on huge image-text collections. At the same time, language models were becoming strong and open. The authors asked a simple question. Can we reuse a strong image encoder and a strong language model as they are, and train only a small connector?

Earlier ideas it built on

• Contrastive image-text learning, as in CLIP, which aligns image and text embeddings.
• Frozen-model prompting, where a vision encoder feeds a frozen language model through a trained mapping.
• Perceiver-style resampling, where a fixed set of latent queries reads from a large input through cross-attention.
• Vision Transformers, which made image encoders look like text transformers.
• Parameter-efficient tuning in the spirit of LoRA and adapters, where a small module adapts a large model.

What came after

BLIP-2 inspired a wave of connector-based assistants. Many later systems replaced the Q-Former with a simpler projection layer and added instruction data, as in LLaVA. Others kept query-based resampling. The debate between "compress with queries" and "project every patch" is still open, and both camps cite BLIP-2 as a reference point.

How it works

BLIP-2 is trained in two stages. Each stage has a different job.

Step 1: encode the image

A frozen image encoder, a large Vision Transformer in the original design, turns the image into a grid of patch features. Think of cutting a photo into small tiles and writing a short description vector for each tile. Because the encoder is frozen, its weights never change. This keeps its general visual knowledge intact and saves memory, since no gradients are stored for it.

Step 2: let the queries look at the image

The Q-Former owns a small set of learned query vectors. The original design used a few dozen. Each query is like a question card that the model learns to write by itself. The queries attend to the image features through cross-attention. After a few layers, each query has gathered the information that is most useful for language.

The key effect is compression. An image may produce hundreds of patch features. The Q-Former reduces them to a short fixed-length output. That fixed length makes the language model side cheap and predictable.

Step 3: share a text path

The Q-Former has two transformer submodules that share self-attention layers. One interacts with the image through the queries. The other works as a text encoder and decoder. Because the queries and the text can attend to each other under different attention masks, the same module can be trained with several objectives.

Stage 1: vision-language representation learning

In the first stage the Q-Former is connected only to the frozen image encoder. It is trained with three objectives at once.

The goal of this stage is to force the queries to extract the visual information that is relevant to text. Without this stage, the queries might keep irrelevant detail.

Stage 2: vision-to-language generative learning

In the second stage the Q-Former is attached to a frozen language model. The query outputs pass through a small linear layer so their size matches the language model input. They are placed in front of the text input as soft visual prompts. The language model is then trained, with its own weights frozen, to predict the caption or answer.

Two language model styles were used in the original work.

• Decoder-only models, which continue the text after the visual prompt.
• Encoder-decoder models, where the visual prompt and a text prefix go to the encoder and the decoder writes the rest.

Step 4: inference

At inference time the pipeline is simple.

1. Resize and normalise the image.
2. Run the frozen image encoder.
3. Run the Q-Former to get the query outputs.
4. Project them into the language model space.
5. Optionally add a text prompt such as a question.
6. Generate text with beam search or sampling.

Why freezing helps

• Memory: no optimizer state for billions of parameters.
• Stability: the language model does not forget how to write fluent text.
• Reuse: you can swap in a newer language model and retrain only the bridge.
• Speed: training touches a small fraction of the total parameters.

Why freezing can hurt

• The language model never learns visual vocabulary the connector cannot express.
• The image encoder cannot adapt to unusual domains such as medical scans or charts.
• Fine detail, such as small text in an image, can be lost during compression.

Inside the Q-Former in more detail

The Q-Former is built from a stack of transformer blocks, initialised from a pre-trained text encoder. Each block has self-attention over the queries. Every other block adds cross-attention to the image features.

• Self-attention lets the queries talk to each other, so they do not all collect the same thing.
• Cross-attention lets each query pull information from the image.
• Feed-forward layers mix information inside each query.
• The text side shares the self-attention weights, so queries and words live in one space.

An analogy helps. Imagine a team of reporters sent to cover one event. Each reporter has a beat. They meet between rounds (self-attention) to avoid duplicating stories, and they visit the scene (cross-attention) to collect facts. At the end, the editor reads only their short notes, not the whole scene.

Why the three losses work together

• The contrastive loss makes the global summary of an image close to its caption. It is cheap and scales to large batches.
• The matching loss uses fine-grained interaction. It is a harder test, because the model must catch small mismatches such as a wrong colour or object.
• The generation loss forces the queries to hold enough detail to rebuild the caption word by word.

Together they push the queries to hold both global meaning and local detail.

Hard negatives

For the matching task, easy negatives teach little. A common technique is to pick negatives that look similar by contrastive score. These hard negatives sharpen the model's sense of what truly matches.

Data cleaning by captioning and filtering

The BLIP family popularised a bootstrapping idea. A captioner writes new captions for web images, and a filter removes noisy pairs. The model then trains on cleaner text. Web alt-text is often vague or wrong, so this step improves quality without paying for human labels.

What the language model sees

From the language model's point of view, the query outputs look like a few extra tokens at the start of the input. It does not know they came from an image. This is why the method works with many language models. It also explains why prompts matter: the visual tokens and the text prompt are read together as one sequence.

Scaling behaviour

Larger frozen components generally give better results, since a stronger image encoder gives richer features and a stronger language model gives better reasoning. The bridge stays small in both cases. This is the main economic argument for the design: the part you train does not grow with the part you reuse.

Variants and follow-ups

• Instruction-tuned variants add a text instruction as input to the Q-Former, so the queries extract information relevant to the instruction.
• Some later models swap the Q-Former for a plain MLP.
• Some add video input by running the bridge over several frames.
• Others add region-level inputs, such as boxes, to point at objects.

Key concepts and terms

A worked example

This example shows how to caption an image and answer a question with the Hugging Face Transformers library. The paths are placeholders. Model identifiers may change, so check the model hub for the current names.

import torch
from PIL import Image
from transformers import Blip2Processor, Blip2ForConditionalGeneration

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if device == "cuda" else torch.float32

model_id = "example-org/blip2-checkpoint"  # placeholder name
processor = Blip2Processor.from_pretrained(model_id)
model = Blip2ForConditionalGeneration.from_pretrained(
    model_id, torch_dtype=dtype
).to(device)

image = Image.open("example_chart.png").convert("RGB")

# 1) Caption: no text prompt
inputs = processor(images=image, return_tensors="pt").to(device, dtype)
out = model.generate(**inputs, max_new_tokens=30)
print(processor.batch_decode(out, skip_special_tokens=True)[0].strip())

# 2) Question answering: add a text prompt
prompt = "Question: what type of chart is this? Answer:"
inputs = processor(images=image, text=prompt, return_tensors="pt").to(device, dtype)
out = model.generate(**inputs, max_new_tokens=20)
print(processor.batch_decode(out, skip_special_tokens=True)[0].strip())

Reading the example

• The processor handles both resizing the image and tokenising the text.
• With no prompt, the model writes a caption.
• With a prompt in a question-and-answer format, the model continues the text.
• Half precision saves memory on a GPU. On a CPU, use full precision.

A numeric walk-through of the compression

Suppose the image encoder produces 256 patch vectors, each of size 1,408. That is about 360,000 numbers. The Q-Former has 32 queries, each of size 768. The output is 32 times 768, which is about 24,600 numbers. The language model receives roughly one fifteenth of the raw feature volume. The numbers here are illustrative, and real sizes depend on the chosen encoder.

Using it for tables rendered as images

A common practical case is a screenshot of a table or a chart. You can ask the model to describe the layout, name the axes, or say what trend is visible. Treat the answer as a hint, not as data. For exact numbers, read the source table directly.

Strengths and limitations

Strengths

• Low training cost, since only the bridge is trained.
• Strong zero-shot behaviour on captioning and question answering when paired with a good language model.
• Modular design. You can swap the image encoder or the language model.
• Compact visual representation, which keeps language model context use small.
• A clear two-stage recipe that is easy to reproduce and study.
• Good foundation for instruction-tuned successors.

Limitations

• Fixed query count limits detail. Dense scenes, small text and fine charts can lose information.
• The language model is frozen, so it cannot learn new visual skills deeply.
• Zero-shot prompts can be brittle. Small wording changes alter the answer.
• It can hallucinate objects that are not in the image, a common failing of vision-language models.
• Weaker at reading text in images than models trained with dedicated document data.
• Multi-image reasoning and long visual context are not its strength.
• Performance depends on the quality and coverage of the pre-training image-text data.

Risks to keep in mind

• Biases present in web-scale image-text data can show up in captions.
• Captions about people can encode stereotypes.
• Answers sound confident even when wrong. Always verify in high-stakes use.

How it compares with alternatives

How to choose

• Pick BLIP-2 when you want a cheap, modular captioner or a retrieval-ready representation.
• Pick a projection-based assistant when you want chat, instruction following and richer detail.
• Pick a contrastive model when you only need similarity search over images and text.
• Pick a task-specific OCR or document model when the main job is reading text in images.

Role in tabular foundation models

BLIP-2 is not a table model, but its design is a useful template for models that learn from raw tables. The link is concrete once you look at the pieces.

The bridge pattern for tables

A tabular foundation model such as TabPFN reads a whole table as input. A chat assistant that must reason about a table needs a similar bridge. A small trained module can compress a table encoder's output into a short set of vectors that a frozen language model can read. This is the same structure as the Q-Former: a frozen perception module, a learned compressor, and a frozen reasoner.

Row and column structure

Images have a grid. Tables have rows and columns. The Q-Former's learned queries can play the role of "column readers" or "row readers". Each query learns to attend to a pattern, such as a numeric trend or a group of related columns, and passes a summary forward. This helps when a table has more cells than any language model context can hold.

Mixed types and missing values

Real tables mix numbers, categories, dates and free text. A query-based compressor can take a heterogeneous set of cell embeddings and still produce a fixed-size output. Missing cells can be represented by a special embedding. The attention then learns to ignore or to flag them. The same idea supports imputation, where a model predicts the missing cell from the others.

No column names

Tables often arrive with anonymous headers such as col_1 or x7. A model that learns from structure alone, not from names, needs to infer meaning from values and relations. Pre-training the compressor with contrastive and matching objectives, as in Stage 1 of BLIP-2, is a way to align table content with descriptions. After alignment, the model can propose a semantic label for an anonymous column. This supports sector and semantic classification, such as deciding that a column holds currency amounts or postal codes.

Pre-training on synthetic and real tables

BLIP-2's first stage uses image-text pairs. A tabular analogue uses table-description pairs. Because real paired data is scarce, synthetic tables with generated descriptions can fill the gap. The foundation model idea, also seen in TabPFN, is to pre-train on a very wide family of synthetic data-generating processes, then apply the model with no further training.

In-context learning

In-context learning means the model solves a new task from examples placed in its input, with no weight updates. BLIP-2's frozen language model already supports this for text. Pairing a table compressor with it lets a user show a few labelled rows and ask the model to label the next. Pure tabular foundation models do this natively inside the table encoder, and a bridge lets a chat model hand off to them.

Continual learning

Frozen backbones fit continual learning well. If the bridge is the only trained part, you can train a new bridge for each new domain, keep the old ones, and avoid overwriting earlier skills. This reduces catastrophic forgetting. See continual learning for how this idea is applied to table models.

Imputation and anomaly detection

• Imputation: compress the observed cells, then decode a prediction for the masked cells.
• Anomaly detection: compare a row's compressed summary against the typical summaries. A row far from the cluster is suspicious.
• Both use the same learned representation, so one bridge can serve several downstream jobs.

Where the analogy breaks

• Images are dense and local. Tables are sparse, permutation-invariant over rows and often over columns.
• Cell values need exact numeric handling that a soft prompt can blur.
• A language model can misread numbers. Keep exact computation in code or in a dedicated tabular model.

For results on tabular tasks, see /model/benchmarks. For the broader research context, see /model/research.

Practical guidance and best practices

Choosing a configuration

• Start with a publicly available pre-trained checkpoint, not a custom training run.
• Match the language model to the job. Instruction-tuned encoder-decoder models are often better at short answers. Larger decoder-only models are often better at fluent captions.
• Check the license of each component, since the image encoder, the language model and the bridge may carry different terms.

Inference tips

• Use half precision or 8-bit loading to fit on smaller GPUs. See bitsandbytes for the idea of low-bit loading.
• Use a consistent prompt format. For question answering, a "Question: ... Answer:" style often works well.
• Limit the output length to avoid rambling.
• Use beam search for stable captions and sampling for varied ones.
• Batch images of similar size to improve throughput.

Fine-tuning tips

• Prefer parameter-efficient methods such as LoRA on the language model or the Q-Former.
• Keep the image encoder frozen unless your images look very different from web photos.
• Build a small, clean, well-labelled set. Quality beats quantity at this scale.
• Hold out a validation set and track both automatic scores and human review.

Evaluation tips

• Use more than one metric. Captioning scores reward n-gram overlap and miss factual errors.
• Test for hallucination by asking about objects that are absent.
• Check robustness with rotated, cropped and low-light images.
• Review failure cases by category, not only by average score.

Deployment tips

• Cache the image features if the same image is queried many times.
• Serve the language model with an efficient engine such as vLLM when throughput matters.
• Log prompts and outputs for audit, and strip personal data from images where required.
• Add a human check for any decision with legal or medical effect.

Prompt patterns that tend to work

A small evaluation checklist

1. Collect 100 to 200 images that look like your real data.
2. Write the expected answer for each, with a second reviewer.
3. Run the model with fixed prompts and fixed decoding settings.
4. Score exact match for short answers and human rating for captions.
5. Add a set of negative questions about absent objects.
6. Record failure categories and fix the biggest one first.

Working with tables and charts

• Export tables as data when you can. Do not photograph them.
• If you must use an image, crop to one table and keep resolution high.
• Ask structural questions, such as column count or axis labels, before value questions.
• Cross-check any numbers against a text extraction.
• Pass confirmed values, not guessed values, to a downstream tabular model.

Cost control

• Run the image encoder once per image and cache the result.
• Use shorter outputs for classification-style tasks.
• Load the language model in low precision and measure the quality drop on your own set.
• Batch requests to keep the GPU busy.

Common mistakes

• Treating a caption as ground truth. Captions are model guesses.
• Using it to read exact numbers from a chart. Use the source data instead.
• Changing the prompt format between training and inference.
• Unfreezing everything on a tiny dataset, which causes overfitting and forgetting.
• Ignoring image preprocessing. A wrong resize or colour order silently hurts quality.
• Comparing models on different prompts, then blaming the model.
• Forgetting that the fixed query count limits detail. Crop and zoom on the region of interest instead.
• Skipping license review of the frozen components.
• Evaluating only on easy photos, then deploying on scans, screenshots or diagrams.
• Expecting a single model to handle tables, charts and photos equally well.

Design trade-offs in one view

• Fewer queries mean a cheaper language model pass but less detail.
• More queries mean richer detail but a longer prompt and slower generation.
• A bigger bridge can learn more but loses the low-cost advantage.
• A smaller bridge is easy to train but may become the bottleneck.

Data and training considerations

• Pre-training uses large collections of image-text pairs gathered from the web, plus synthetic captions.
• Image resolution affects the number of patches, so higher resolution costs more in the encoder.
• Augmentation such as random crops helps robustness, but cropping can remove the object named in the caption.
• Training with mixed precision is standard and keeps memory in check.
• Because the large parts are frozen, training runs are short compared with end-to-end models.

Typical use cases

• Alt-text suggestions for accessibility, reviewed by a person.
• Cataloguing photo libraries with searchable descriptions.
• First-pass tagging of product images in a retail catalogue.
• Visual question answering for support tools, with human review.
• Generating text-image pairs to bootstrap a smaller specialised model.
• Describing figures in scientific or business documents as a rough index.

Questions to ask before adopting it

1. Is a caption or short answer enough, or do I need exact extraction?
2. Do my images look like web photos, or are they scans and diagrams?
3. Who reviews wrong answers, and what do they cost?
4. Are the licences of all three components compatible with my use?
5. Can I afford a GPU for serving, or must I run on CPU?

Relation to nearby ideas

• Adapters and prompt tuning also train a small piece next to a frozen model.
• Retrieval-augmented generation feeds a language model external text, while BLIP-2 feeds it visual summaries.
• Distillation could shrink the combined stack into a smaller captioner once quality is acceptable.
• Quantisation, as in GPTQ, can reduce the language model size for deployment.

Reading the research

For the lineage of table models and how bridging ideas are tested, see /model/research. For how a table encoder is organised internally, see /model/architecture. For measured results on tabular tasks, see /model/benchmarks.

Frequently asked questions

What does BLIP-2 do?

BLIP-2 reads an image and produces text about it. It can write captions, answer questions about the picture and support matching images to text. It does this by connecting a frozen image encoder to a frozen language model through a small trained module called the Q-Former.

What is the Q-Former?

The Q-Former is a small transformer that holds a fixed set of learned query vectors. Those queries look at the image features through cross-attention and return a short summary. The summary is passed to the language model as a soft prompt, so the language model can use visual information without being retrained.

Why are the image encoder and language model frozen?

Freezing keeps their existing knowledge safe and makes training cheap. Only the bridge needs gradients and optimizer state, so the memory and compute costs are far lower than end-to-end training. It also lets teams swap in a newer component later and retrain only the connector.

How is BLIP-2 different from LLaVA?

BLIP-2 compresses image features into a short set of query outputs and trains mainly that bridge. LLaVA projects many patch features into the language model and then fine-tunes on visual instruction data. BLIP-2 tends to be cheaper and more modular, while LLaVA-style models tend to be better at chat and detailed instruction following.

Can BLIP-2 read text inside images?

It can read some large or clear text, but it is not designed as a dedicated document reader. The compression step can drop small characters. For invoices, forms and dense tables, a document-focused model or a classic OCR step is usually more reliable.

Is BLIP-2 a tabular foundation model?

No. It works on images and text. Its design pattern is still relevant, because a small trained bridge between a frozen perception model and a frozen reasoning model is a common way to connect table encoders to language models.

Can I fine-tune BLIP-2 on my own images?

Yes. The usual path is parameter-efficient fine-tuning, such as low-rank adapters on the language model or the Q-Former, while keeping the image encoder frozen. Use a small, clean dataset, hold out a validation set and check for hallucination after training.

Does BLIP-2 hallucinate?

Yes, like other vision-language models it can mention objects that are not present or miss objects that are. Test it with questions about absent items, use clear prompts, and keep a human in the loop where errors are costly.

What hardware does it need?

The size depends on the chosen language model. Smaller variants run on a single consumer GPU in half precision. Larger ones need more memory, and low-bit loading can help. Training only the bridge needs far less memory than training the whole stack.

Where does BLIP-2 fit among multimodal models?

It sits at the efficient, modular end. It is a good baseline for captioning and retrieval, a good teaching example of the connector pattern, and a stepping stone toward instruction-tuned multimodal assistants.

Glossary

• Adapter: A small trainable module added to a large frozen model.
• Attention mask: A rule that decides which tokens can see which others.
• Beam search: A decoding method that keeps several candidate sentences at once.
• Bootstrapping: Using a model to produce or clean data to train itself or its successor.
• Captioning: Writing a short description of an image.
• Contrastive learning: Training by pulling matching pairs together and pushing others apart. See contrastive learning.
• Cross-attention: Attention from one set of vectors to another set.
• Decoder-only model: A language model that predicts the next token from earlier ones.
• Encoder-decoder model: A model that reads input with an encoder and writes output with a decoder.
• Frozen: Not updated during training.
• Hallucination: Output that is fluent but not supported by the input.
• Learned query: A trainable vector used to pull information from another input.
• Patch: A small tile of an image treated as one token.
• Projection layer: A linear layer that maps vectors to another size.
• Soft prompt: A learned vector sequence used like a prompt.
• Zero-shot: Performing a task with no examples of that task in training.

Summary

BLIP-2 connects a frozen image encoder to a frozen language model with a small trained Q-Former. Learned queries read the image through cross-attention and hand a compact summary to the language model. A two-stage recipe first teaches the queries what text cares about, then teaches the language model to use them.

The method is cheap, modular and effective for captioning, question answering and retrieval, but it loses fine detail and can hallucinate. For tabular work, its main value is the pattern: freeze strong components, train a thin bridge, and reuse the bridge for imputation, classification and anomaly checks. Related reading includes LLaVA, CLIP, Vision Transformers and /model/architecture.

Related

It belongs to Vision and multimodal in my stack, under Foundation models and applied ML. Others in the same group: