Multimodal models
Multimodal models understand and generate more than one kind of data, such as text, images, audio and video. Learn how they combine modalities and where they help.
Overview
A multimodal model is an AI model that works with more than one kind of data, called a modality. The common modalities are text, images, audio and video, but tables, sensor readings and 3D data also count. A multimodal model might read a photo and a question and answer in text, listen to speech and reply in speech, or take a text description and produce an image. The key feature is that the different inputs meet inside one model, so it can relate what it sees to what it reads or hears.
This matters because real information is rarely text alone. Invoices are scanned images with tables. Meetings are audio with slides. Support requests arrive with screenshots. A text-only model needs a separate tool for each format and loses detail in conversion. A multimodal model can look at the screenshot itself, read the chart directly and notice that the graph contradicts the caption.
Think of a doctor examining a patient. They read the notes, look at the X-ray, listen to the chest and hear the patient describe the pain. Each source alone is incomplete; together they make a diagnosis. A multimodal model tries to combine sources in the same way, instead of asking a specialist for each sense and stitching the reports together.
What it is
Most current multimodal systems that answer in text, often called vision-language models (VLMs), follow a common pattern.
| Part | Role |
|---|---|
| Modality encoder | Turns raw input such as pixels or sound into vectors, for example a vision transformer |
| Projector or adapter | Maps those vectors into the language model's embedding space |
| Language model backbone | A transformer that reasons over text and projected tokens together |
| Output head or decoder | Produces text, or hands off to an image or audio generator |
| Tokeniser for each modality | Splits input into units, such as image patches or audio frames |
| Training data | Paired data, such as images with captions or speech with transcripts |
Some models are built by joining separate pre-trained parts with a small adapter. Others are trained as one network on mixed data from the start, often described as natively multimodal.
Where it came from
Early multimodal research combined separate vision and language systems, for example image captioning with convolutional networks and recurrent decoders in the mid-2010s. The transformer era changed the picture. The Vision Transformer in 2020 showed that images could be split into patches and processed like tokens.
In January 2021 OpenAI introduced CLIP, which trained an image encoder and a text encoder together on image and caption pairs so that matching pairs had similar embeddings. CLIP became a standard vision encoder for later systems. DeepMind's Flamingo in 2022 connected a frozen vision encoder to a frozen language model and could answer questions about images with only a few examples. In 2023 Salesforce's BLIP-2 used a light querying module to bridge vision and language, and LLaVA showed that a simple projection layer plus visual instruction tuning could produce a capable open assistant.
Commercial models followed. OpenAI's GPT-4, announced in March 2023, accepted image inputs, and vision access widened later that year. Google introduced Gemini in December 2023 as a model trained to be multimodal from the start. In May 2024 OpenAI released GPT-4o, handling text, images and audio in one model. Speech models such as Whisper and image generators based on diffusion models developed alongside. Today most frontier models accept images and documents, and many handle audio and video.
How it works
Step 1: Encode each modality
An image is resized and cut into small patches. A vision encoder turns each patch into a vector. Audio is converted into a spectrogram or frames and encoded similarly. Video is usually handled as a sequence of sampled frames, sometimes with audio.
Step 2: Project into a shared space
A projector, often a small neural network, maps the encoder's vectors into the same space as the language model's word embeddings. After this step, image patches look to the language model like extra tokens.
Step 3: Interleave with text
The projected tokens are placed in the sequence alongside the text tokens, for example "[image tokens] What is the total on this invoice?". The language model attends across all of them.
Step 4: Train in stages
A typical recipe first trains only the projector on image and caption pairs to align the spaces. A second stage fine-tunes more of the model on instruction data, such as questions about images with good answers. Natively multimodal models instead mix modalities throughout pre-training.
Step 5: Generate output
For text output, the language model writes as usual. For image or audio output, the model either produces special tokens that a decoder turns into pixels or sound, or calls a separate generator such as a diffusion model.
Step 6: Manage token budgets
Images and video produce many tokens. For example, a 336 by 336 pixel image cut into 14 by 14 pixel patches gives 576 patch tokens. High-resolution documents and long videos multiply this, so models use tiling, pooling or frame sampling to control cost.
Key concepts and terms
| Term | Meaning |
|---|---|
| Modality | A type of data, such as text, image, audio or video |
| Vision-language model | A model that takes images and text and outputs text |
| Patch | A small square of an image treated like a token |
| Projector | A layer that maps encoder outputs into the language model's space |
| Contrastive pre-training | Learning to match pairs, as in CLIP |
| Visual instruction tuning | Fine-tuning on questions and answers about images |
| Natively multimodal | Trained on several modalities from the start |
| Grounding | Linking words to specific regions or moments in the input |
| OCR | Optical character recognition, reading text in images |
| Any-to-any | Models that can take and produce several modalities |
A worked example
An illustrative accounts team processes 2,000 supplier invoices a week in mixed formats: PDFs, phone photos and scanned faxes.
Old pipeline: OCR software extracts text, then rules and a text model pull out supplier, date, total and VAT. Tables often come out scrambled, and about 15 per cent of invoices need manual fixing.
New pipeline:
- Each invoice page is sent as an image to a vision-language model with the instruction: "Extract supplier name, invoice date, line items and total as JSON matching this schema."
- The model reads the layout directly, so table rows stay together and handwritten notes are seen in context.
- A validation step checks that line items add up to the total and that the date is valid.
- Invoices that fail validation, about 4 per cent in this example, go to a person.
- Each page uses about 1,500 image tokens plus 300 text tokens. The team tracks cost per invoice and latency.
In this illustrative case the manual-fix rate drops from 15 to about 4 per cent. The gain comes from the model seeing layout and text together, plus a simple arithmetic check that catches errors.
Strengths and limitations
Strengths:
- Understands layout, charts, screenshots and photos directly.
- Removes fragile conversion steps such as separate OCR.
- Enables natural voice interaction and visual question answering.
- One model can cover many input types, simplifying systems.
- Useful for agents that operate graphical interfaces; see computer use.
Limitations:
- Images, audio and video consume many tokens, raising cost and latency.
- Can misread small text, count objects poorly or confuse spatial relations.
- May describe things that are not in the image, a visual form of hallucination.
- Long video understanding is still limited and expensive.
- Harder to evaluate, since outputs depend on visual detail.
- Privacy concerns increase with photos, voices and faces.
How it compares with alternatives
| Approach | How it handles non-text data | Best for |
|---|---|---|
| Multimodal model | Encodes and reasons over all inputs together | Mixed documents, visual Q&A, voice |
| Pipeline of specialists, for example OCR plus text model | Converts each input to text first | Clean, high-volume formats |
| Contrastive embedding models such as CLIP | Shared vectors for search and matching | Image search, zero-shot labels |
| Task-specific vision models such as YOLO | Fixed outputs such as boxes or labels | Fast, precise detection |
| Speech-only models such as Whisper | Transcribes audio to text | Transcription at scale |
Practical guidance and best practices
- Start with the modality that carries the most signal; do not add audio or video just because you can.
- Measure token use per input and set resolution or frame-rate limits accordingly.
- Ask for structured output and validate it with simple rules, such as totals adding up.
- Crop or tile large documents so small text is legible to the model.
- Build an evaluation set of real, messy inputs, including poor photos and scans.
- Keep a fast specialist model for narrow, high-volume steps such as detection.
- Handle faces, voices and personal documents under clear privacy rules.
- Ask the model to cite page numbers or regions so people can check its answers.
Common mistakes
- Sending full-resolution images when a smaller crop would do, wasting tokens.
- Trusting extracted numbers without arithmetic or format checks.
- Testing only on clean, digital documents when real inputs are phone photos.
- Asking for precise counts or measurements and assuming they are exact.
- Ignoring that video is sampled, so short events between frames may be missed.
Frequently asked questions
What is the difference between a multimodal model and a vision-language model?
A vision-language model is one kind of multimodal model, handling images and text. "Multimodal" is the broader term and also covers audio, video and other data types.
Do multimodal models replace OCR?
For many document tasks they can, because they read text and layout together. For very high volumes of clean documents, dedicated OCR can still be cheaper and more predictable. Many teams combine the two.
How are images turned into tokens?
The image is split into patches, each patch is encoded into a vector by a vision encoder, and a projector maps those vectors into the language model's input space, where they are treated like word tokens.
Can these models generate images too?
Some can, either natively or by calling an image generator such as a diffusion model. Many vision-language models only understand images and output text.
Why are multimodal requests expensive?
Images, audio and video turn into many tokens. A single high-resolution page can cost as much as several pages of text, and video multiplies that by the number of frames.
Are multimodal models safe to use with personal photos?
They raise extra privacy questions because images and audio can reveal identity and sensitive details. Use clear consent, data minimisation and provider terms that match your obligations.
Glossary
- Encoder: a network that converts raw input into vectors.
- Embedding space: the vector space in which tokens are represented.
- Spectrogram: a picture-like representation of sound over time and frequency.
- Frame sampling: selecting some video frames to process.
- Hallucination: output that is not supported by the input.
- Adapter: a small trainable module connecting larger frozen parts.
- Resolution: the number of pixels in an image.
- Alignment training: teaching separate encoders to share a common space.
Summary
Multimodal models combine several data types, most often text and images, and increasingly audio and video, inside one model. Building on the Vision Transformer and CLIP, systems such as Flamingo, BLIP-2 and LLaVA showed how to connect vision encoders to language models, and commercial models from 2023 onwards made image and audio input standard. They remove fragile conversion steps and understand layout and visuals directly, but they cost more tokens, can misread fine detail and need careful evaluation on realistic inputs.
Related
It belongs to LLM concepts in my stack, under Generative AI and LLMs. Others in the same group: