LLaVA

Overview

LLaVA is an open vision-language assistant. It connects a pre-trained vision encoder to a pre-trained language model with a simple projection layer, then fine-tunes the result on visual instructions. After this training, you can send it a picture and ask questions in plain language, and it answers in conversation.

The name stands for Large Language and Vision Assistant. Its main contribution is a recipe rather than a new architecture: use existing parts, build good instruction data, and train in two short stages. The recipe made multimodal chat accessible to research groups and developers who could not afford to train a model from scratch. The same recipe, pairing a perception encoder with a language model through a thin connector, is now common when people build assistants that read documents, screenshots and tables. See BLIP-2 for a related but different connector design.

What it is

LLaVA is both a model family and a training method called visual instruction tuning. It has three building blocks.

• A vision encoder, usually a CLIP-style Vision Transformer, that turns an image into patch features.
• A connector, which began as a single linear layer and later became a small multilayer perceptron.
• A language model, which reads the projected image features together with the user's text and writes a reply.

Everyday analogy

Picture a person who is very good at conversation but cannot see, and a friend who can see but speaks only in short labels. You teach the friend to whisper what they see in a form the first person understands, and you practise with many example conversations. The projection layer is the whisper. The instruction data is the practice.

What it can do

• Describe an image in detail.
• Answer questions about objects, colours, positions and activities.
• Follow instructions such as "list the items on the table".
• Hold a multi-turn conversation about one image.
• Give reasoning about a scene, such as explaining why something is unusual.

What it is not

• It is not an image generator. It reads images, and it writes text. For generation, see Stable Diffusion.
• It is not a guaranteed OCR engine. It can read some text but may fail on dense pages.
• It is not a single checkpoint. Many versions exist, with different language models and image resolutions.

Where it came from

LLaVA appeared in 2023 from academic researchers. It arrived soon after open language models became widely available and after instruction-tuned chat models showed how much behaviour changes with good instruction data.

The ideas it combined

• Contrastive image-text pre-training, which gave strong vision encoders such as CLIP.
• Open instruction-tuned language models fine-tuned from base models.
• Instruction tuning, where a model learns to follow commands from examples of commands and answers.
• Earlier multimodal bridges such as BLIP-2 and Flamingo-style models.
• Using a stronger text-only model to write training data from image descriptions.

The data trick

Good image-instruction data was scarce. The authors used a strong text-only language model to write conversations, detailed descriptions and reasoning questions. The model was given not the image itself but its caption and the object boxes, as plain text. From these it produced realistic questions and answers. This created a large instruction set without manual labelling.

Later versions

Follow-up versions improved the connector, raised image resolution, added more academic task data and supported more language model backbones. Some variants handle multiple images, video, or very high resolution by tiling an image into several crops. Details differ by version, so check the specific model card before relying on a feature.

Why it mattered

LLaVA showed that a small amount of well-designed instruction data and a simple connector can give strong multimodal chat. It lowered the barrier to entry and became a base for many community models and research studies.

How it works

The pipeline can be described as four steps for inference and two stages for training.

Step 1: encode the image

The vision encoder splits the image into fixed-size patches, like cutting a photo into a grid of postage stamps. Each patch becomes a feature vector. The encoder was pre-trained to match images with text, so its features already carry rich meaning.

Step 2: project into word space

The language model expects vectors of a certain size, in a space shaped by words. The projection layer maps each patch feature into that space. After projection, each patch looks like a "visual word" to the language model.

Step 3: build one sequence

The visual tokens and the text tokens are placed in one sequence. A typical layout puts a system message first, then the image tokens, then the user question. The language model reads all of this with ordinary self-attention.

Step 4: generate the answer

The language model predicts the answer one token at a time. Because it has been tuned on conversations, it responds in a helpful style. The image tokens act as extra context, similar to a pasted document.

Training stage 1: feature alignment

Only the projection layer is trained. The vision encoder and the language model are frozen. The data is simple image-caption pairs. The task is to produce the caption given the image. This teaches the connector to translate visual features into the language model's space.

Training stage 2: end-to-end instruction tuning

The projection layer and the language model are trained together, while the vision encoder usually stays frozen. The data is the visual instruction set: conversations, detailed descriptions and reasoning questions. Loss is computed only on the answer tokens, not on the question tokens. This teaches the model to follow instructions about images.

Why two stages

• If you train everything at once from a random connector, the noisy signal from an untrained bridge can damage the language model.
• Aligning first gives a stable starting point.
• Stage 2 then focuses on behaviour, not on basic alignment.

Image resolution and token count

Each patch becomes a token, so a higher resolution image costs more tokens and more compute. A typical encoder produces a few hundred patch tokens. Doubling the side length of the image multiplies the token count by about four. Newer variants handle this by splitting a large image into tiles and encoding each one, which gives detail at a higher cost.

Compare with a query-based bridge

BLIP-2 compresses the image into a fixed small set of query outputs. LLaVA keeps one token per patch. The second approach preserves more spatial detail and is simpler. It also uses more of the language model's context window. Neither is strictly better. The right choice depends on the task, the budget and the context length.

Quantised and efficient serving

Once trained, the language model dominates the memory cost. Teams often load it in lower precision, using ideas from bitsandbytes or GPTQ. Inference engines such as vLLM and llama.cpp can serve multimodal checkpoints on suitable hardware. Support varies by model and engine version.

Inside the instruction data

The original instruction set had three flavours of sample.

Later versions mixed in academic datasets for visual question answering, region reference and text recognition, with short-answer prompts that tell the model to answer in a single word or phrase. That mix helps benchmark scores, and it also changes style, so test the behaviour you actually need.

How loss is computed

During instruction tuning, the sequence has system text, image tokens, the user turn and the assistant turn. The model predicts every next token, but the loss ignores everything except the assistant's reply. This is like grading a student only on the answer, not on copying the question. It stops the model from wasting capacity on memorising prompts.

Connector choices

• A single linear layer is the simplest option. It is a plain matrix multiplication.
• A two-layer MLP with a nonlinearity gives the connector more flexibility and was found to help in later versions.
• Pooling or merging patch tokens reduces the count, at the cost of detail.
• Query-based resamplers, as in BLIP-2, compress to a fixed number of tokens.

Resolution strategies

1. Resize everything to one square size. Simple, but distorts wide images.
2. Pad to a square. Keeps shape, wastes tokens on blank space.
3. Tile the image into a grid of crops plus a low-resolution overview. Keeps detail, costs more tokens.
4. Use an encoder that accepts variable sizes. Flexible, needs matching position handling.

Typical failure patterns

• Inventing a plausible object because it often appears in similar scenes.
• Mixing up left and right, or above and below.
• Reading a number wrongly by one digit.
• Answering a yes or no question with yes by default.
• Describing what is usual instead of what is shown.

Most of these come from the language model's prior beliefs overpowering weak visual evidence. Better resolution, clearer prompts and negative training examples reduce them.

Where LLaVA-style models are used

• Assistants that describe screenshots and interfaces.
• Research on grounded dialogue and reasoning.
• Content moderation support, with human review.
• Cataloguing and tagging of image collections.
• Education tools that explain diagrams.
• Prototype tools that answer questions about charts and dashboards.

Measuring progress

Evaluation uses a mix of benchmarks for visual question answering, science questions with diagrams, text reading in images, and open-ended chat judged by a stronger model. Each benchmark has blind spots, so no single number is enough. For tabular models the equivalent evidence is on /model/benchmarks.

Key concepts and terms

A worked example

This example uses the Hugging Face Transformers library to ask a question about an image. Model names and processor classes can change between releases, so use a current model card as the reference.

import torch
from PIL import Image
from transformers import AutoProcessor, LlavaForConditionalGeneration

model_id = "example-org/llava-checkpoint"  # placeholder name
processor = AutoProcessor.from_pretrained(model_id)
model = LlavaForConditionalGeneration.from_pretrained(
    model_id, torch_dtype=torch.float16, device_map="auto"
)

image = Image.open("example_dashboard.png").convert("RGB")

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "List the metrics shown and their trends."},
        ],
    }
]
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(images=image, text=prompt, return_tensors="pt").to(model.device)

out = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(processor.decode(out[0], skip_special_tokens=True))

Reading the example

• The chat template inserts special tokens that mark where the image goes.
• Setting do_sample=False gives a repeatable answer.
• The same code with a different question gives a different task, such as describing or counting.

Building your own instruction data

A small example of one training record in a common conversation format.

{
  "image": "images/example_0001.png",
  "conversations": [
    {"from": "human", "value": "<image>\nWhat does this chart compare?"},
    {"from": "gpt", "value": "It compares monthly counts for two product lines."}
  ]
}

A numeric walk-through of token cost

Suppose an image yields 576 patch tokens, and your text prompt has 40 tokens. The language model sees 616 tokens before it writes anything. With a 4,096-token context, about 3,480 tokens are left for the answer and any further turns. If you add a second image of the same size, the visual part alone is 1,152 tokens. The numbers are illustrative, and real counts depend on the encoder and its resolution.

Fine-tuning on your own data

1. Collect images and write or generate question-answer pairs.
2. Review a sample by hand to remove wrong answers.
3. Keep the vision encoder frozen.
4. Train with a low-rank method such as LoRA or QLoRA on the language model, and train the projection fully.
5. Evaluate on held-out images with fixed prompts.
6. Check for new hallucinations, not only for gains.

Strengths and limitations

Strengths

• Simple, understandable design: encoder, projector, language model.
• Strong conversational behaviour from instruction tuning.
• Open ecosystem, with many checkpoints, adapters and tools.
• Cheap to adapt, especially with parameter-efficient fine-tuning.
• Keeps spatial detail, since each patch is a token.
• Easy to extend with new backbones as better parts appear.

Limitations

• Token cost grows quickly with image size and with more than one image.
• It can hallucinate objects, text or numbers that are not present.
• Reading small or dense text is unreliable at low resolution.
• Counting and precise spatial reasoning are weak points.
• Quality depends heavily on the instruction data, which can contain errors written by another model.
• Biases in the vision encoder and in the language model carry through.
• It may refuse or ignore details if the prompt is vague.

Safety and licensing notes

• Check the licence of the vision encoder, the language model and the data.
• Do not rely on the model for identification of people.
• Add review steps for medical, legal or financial uses.
• Treat the model's explanation as a claim to verify, not as evidence.

How it compares with alternatives

How to choose

• Use LLaVA-style models for open, adaptable image chat.
• Use a query-based bridge when context length is tight.
• Use contrastive models when you only need similarity.
• Use an OCR or document model when the text on the page is the main content.

Role in tabular foundation models

LLaVA is a vision model, yet its method maps cleanly to work on raw tables. The shared theme is how a language model gains a new input type without being rebuilt.

Connector pattern for tables

Replace the vision encoder with a table encoder. The encoder reads cells, rows and columns and outputs a set of embeddings. A projection layer maps them into the language model space. Then the assistant can chat about the table. This follows the same two-stage logic: first align table embeddings with simple text descriptions, then instruction-tune on questions and answers about tables.

Row and column structure

A patch grid has two spatial dimensions. A table also has two, but with different meaning. Rows are usually examples and columns are usually features, and the order of rows often does not matter. A table connector should respect this. Position encodings should mark column identity, not left-to-right order alone, and the model should not depend on row order.

Mixed types and missing values

Tables mix numbers, categories, dates and text. A good table encoder embeds each type with a suitable method, for example a numeric encoder for numbers and a token embedding for categories. Missing values get a dedicated marker, so the model knows a cell is absent and not zero. When these features reach the language model, the assistant can answer questions such as "which columns have many gaps" and can propose imputations.

No column names

Many real tables have headers like f1, f2 or none at all. The assistant must then infer meaning from values, ranges and relations between columns. Instruction tuning can teach it to propose a semantic label, such as "looks like a date of birth" or "looks like a currency amount". This is a form of semantic and sector classification: assigning a column, or a whole table, to a type or an industry domain from content alone.

Pre-training on synthetic and real tables

LLaVA's instruction data came from a text model rewriting captions. The tabular analogue is to generate tables from synthetic data-generating processes, then have a text model or a program write questions and answers about them. Models such as TabPFN show that pre-training on very many synthetic tables can give a model that works on real ones with no training. Real tables add realism, and synthetic tables add coverage and control.

In-context learning

LLaVA-style assistants can follow examples placed in the prompt. For tables, this allows few-shot labelling: show a handful of labelled rows, then ask for the label of a new row. Tabular foundation models do this inside the encoder with attention over the example rows, without any weight update. An assistant can call such a model as a tool and explain the result in words.

Continual learning

Because the connector is small, it is practical to keep several of them. A new domain gets a new connector or adapter, and old ones remain untouched. This limits forgetting. It parallels the way a tabular model can be updated with new tables while keeping earlier skills.

Imputation and anomaly detection

• Imputation: the assistant describes missing patterns and a table model fills the cells.
• Anomaly detection: the table encoder gives a row embedding, and unusual rows stand out as outliers. The assistant then explains why a row looks odd, using the columns that deviate most.

Honest limits

• Language models are poor at exact arithmetic on many numbers. Keep calculation in code.
• Long tables exceed the context window. Compress with a bridge or sample rows.
• Never treat a chat answer about a table as a verified fact without a check on the source.

For model internals see /model/architecture. For evidence on tabular tasks see /model/benchmarks. For background see /model/research.

Practical guidance and best practices

Choosing a model

• Pick a version with a language model size that fits your hardware.
• Prefer checkpoints with clear model cards, licences and documented prompt formats.
• Test two or three candidates on your own images before committing.

Prompting

• Be specific. "List all visible product names" works better than "What is this?".
• Ask for a format, such as a bullet list or JSON, when you need structured output.
• Ask one thing at a time for hard questions.
• Add "if not visible, say not visible" to reduce guessing.
• Keep temperature low for factual tasks.

Image handling

• Use clear, well-lit images and avoid heavy compression.
• Crop to the region of interest. Zoom helps small text.
• For large pages, tile the image or use a version that supports high resolution.
• Keep colour order and normalisation consistent with the processor.

Training and adaptation

• Start with parameter-efficient methods such as LoRA or QLoRA.
• Mix your data with a general instruction set, to avoid losing general skills.
• Use a validation set that reflects the real mix of images.
• Track hallucination rate separately from accuracy.
• Version your data, prompts and checkpoints.

Serving

• Batch similar requests.
• Cache image features for repeated queries.
• Use an efficient engine such as vLLM when it supports your model.
• Set maximum output length and timeouts.
• Log inputs and outputs while respecting privacy rules.

Evaluation

• Combine automatic benchmarks with human review.
• Test with negative questions about absent objects.
• Check robustness to rotation, blur and crops.
• Slice results by image type and question type.

Project structure that works

• Keep raw images, annotations and generated data in separate folders.
• Store prompts and chat templates in version control with the model checkpoint.
• Record the processor version, since resizing rules can change.
• Keep a frozen evaluation set that is never used for tuning.
• Write a short model card for each fine-tuned adapter.

Data quality rules

1. Remove duplicate and near-duplicate images.
2. Remove answers that mention things not in the image.
3. Balance question types so one style does not dominate.
4. Include questions whose right answer is "not visible".
5. Keep a record of where each sample came from.

Privacy and governance

• Remove personal data from images before sending them to any service.
• Check consent and licence for every image set.
• Keep access logs and a deletion process.
• Review outputs for sensitive attributes before using them in decisions.

Common mistakes

• Trusting a detailed description as if it were a measurement.
• Asking for exact counts or exact numbers and accepting the answer.
• Changing the chat template between training and inference.
• Putting the image after a long text prompt when the template expects it first.
• Fine-tuning on a small, noisy set until the model forgets general skills.
• Using a high-resolution image without checking the token budget.
• Ignoring the licences of the underlying encoder and language model.
• Evaluating on the same images used for tuning.
• Using generated instruction data without sampling it for errors.
• Expecting the model to read dense tables from a photograph. Use the source data instead.

A short debugging checklist

1. Does the processor version match the checkpoint?
2. Is the image token placeholder present in the prompt?
3. Is the dtype consistent across the processor outputs and the model?
4. Is the maximum sequence length large enough for the image tokens?
5. Does a known simple image give a sensible answer?
6. Does greedy decoding give the same result twice?

A quick decision guide

Final checks before release

1. Run the full evaluation set with fixed decoding.
2. Compare against the base checkpoint to confirm there is no regression.
3. Review twenty random outputs by hand.
4. Confirm licences and attribution text.
5. Document known failure modes for users.
6. Plan how you will collect feedback and update the model.

Frequently asked questions

What is LLaVA?

LLaVA is an open vision-language assistant that lets a language model reason about images. It joins a vision encoder to a language model through a projection layer and is fine-tuned on visual instructions, so it can describe pictures and answer questions in a chat.

How is LLaVA trained?

Training has two stages. First, only the projection layer is trained on image-caption pairs to align visual features with the language model. Second, the projection layer and the language model are tuned together on visual instruction data, usually with the vision encoder frozen.

What is visual instruction tuning?

Visual instruction tuning means fine-tuning a model on examples where a user gives an instruction about an image and the model gives a helpful answer. It teaches the model to follow commands, not just to caption, and it is the main reason LLaVA behaves like a chat assistant.

How does LLaVA differ from BLIP-2?

LLaVA turns every image patch into a token and trains with instruction data, which keeps detail and gives good chat behaviour. BLIP-2 compresses the image into a small set of query outputs and trains mainly a bridge between frozen models, which is cheaper but loses some detail.

Can LLaVA read text in images?

It can read clear, large text, and later versions with higher resolution do better. It is not a dedicated OCR system, so small or dense text can be misread. For exact text extraction, use an OCR tool or a document-focused model.

Does LLaVA generate images?

No. It takes images as input and produces text. Image generation needs a different type of model, such as a diffusion model.

Can I run LLaVA on my own computer?

Smaller versions can run on a single modern GPU, especially with lower-precision loading. Larger ones need more memory. Some engines can run quantised versions on a CPU, with lower speed. Check the model card for memory needs.

Is LLaVA useful for tabular data?

Directly, only for tables that appear as images, and even then exact data should come from the source. Indirectly, its method of connecting an encoder to a language model through a projector is a pattern for assistants that explain and query tables.

How do I reduce hallucinations?

Use clear images, specific prompts and low temperature, and tell the model to say when something is not visible. Add negative examples to fine-tuning data, and verify important answers against another source. No setting removes hallucination completely.

What licence applies to LLaVA?

The licence depends on the specific checkpoint, because it combines a vision encoder, a language model and training data that each carry their own terms. Read the model card and the licences of every component before commercial use.

Glossary

• Alignment stage: The first training stage, which teaches the connector using captions.
• Backbone: The main pre-trained network used inside a larger model.
• Chat template: The text format that marks roles in a conversation.
• Context window: The maximum number of tokens a model can read at once.
• Fine-tuning: Continuing training on a smaller, task-specific dataset.
• Greedy decoding: Always choosing the most likely next token.
• Instruction tuning: Training a model to follow commands.
• Low-rank adaptation: Training small added matrices instead of all weights. See LoRA.
• Multimodal model: A model that handles more than one type of input.
• Patch: A small tile of an image.
• Projector: The layer that maps image features to language model space.
• Quantisation: Storing weights with fewer bits to save memory.
• Token: A unit of text or a visual feature treated as one input element.
• Vision Transformer: An image encoder built on transformer blocks. See ViT.
• Zero-shot: Doing a task with no examples of it in training.

Summary

LLaVA is an open assistant that adds vision to a language model through a vision encoder, a simple projector and visual instruction tuning. It trains in two stages: align the connector on captions, then tune on conversations about images. The result is a model that can describe, question and reason about pictures in a chat.

It is simple, adaptable and widely supported, but it uses many tokens, can hallucinate and is not a reliable OCR tool. For tabular work, the same recipe suggests how to attach a table encoder to a language model so it can explain columns, propose semantic labels, support imputation and flag anomalies, while exact calculation stays in code. Related reading includes BLIP-2, CLIP, TabPFN and /model/research.

Related

It belongs to Vision and multimodal in my stack, under Foundation models and applied ML. Others in the same group: