# Vision-language-action models

Vision-language-action models map camera images and a spoken or written instruction to robot actions. Learn the architecture, training data, key models and limits.

## Overview

A vision-language-action model, usually shortened to VLA, is a neural network that looks at the world through cameras, reads an instruction such as "put the red cup in the sink", and outputs the motor commands a robot needs to do it. It joins three abilities in one model: seeing, understanding language, and acting.

Most VLAs are built on top of a vision-language model, the kind of [multimodal model](/tools/multimodal-models) that can describe a picture or answer questions about it. That backbone has already learned a great deal from web images and text: what a cup is, what "red" means, where sinks usually are. The VLA adds the missing piece by training on recordings of robots doing tasks, so that the model learns to turn its understanding into movement. The hope is that web knowledge transfers, so a robot can handle objects and instructions it never saw during robot training.

An everyday picture: think of a new kitchen assistant who has read every cookbook and seen thousands of cooking videos but has never held a knife. They know what "dice the onion" means and can recognise an onion instantly. What they lack is the feel of the movement. A few weeks of supervised practice turns that knowledge into skill. A VLA follows the same path: broad knowledge from the web, then practice from robot demonstrations.

## What it is

| Part | Role | Common choices |
|---|---|---|
| Vision encoder | Turns camera images into tokens the model can reason over | [Vision transformer](/tools/vit) encoders such as SigLIP or DINOv2 |
| Language backbone | Processes the instruction and image tokens together | A pre-trained language model, often a few billion parameters |
| Robot state input | Supplies joint angles, gripper state and similar readings | Projected into tokens or appended to the context |
| Action head | Produces the actions | Discrete action tokens, or a continuous head using diffusion or flow matching |
| Action chunking | Predicts several future actions at once | Chunks of tens of steps executed at high frequency |
| Training data | Demonstrations pairing observations, instructions and actions | Teleoperation recordings, pooled multi-robot datasets, simulation, human video |

Two design choices define a VLA. The first is how actions come out. Early models such as RT-2 and OpenVLA turn each continuous action dimension into one of a fixed number of bins and treat the bins as words, so the language model "writes" actions like text. Later models such as pi0 attach a separate continuous action module trained with flow matching, which suits smooth, fast, dexterous motion. The second is whether one network does everything or whether a slow "thinking" model guides a fast "acting" model. Single-network designs are simpler; dual-system designs, used by Figure's Helix and NVIDIA's GR00T N1, separate reasoning at a few updates per second from control at much higher rates.

## Where it came from

Robot learning from demonstrations is decades old, but the VLA idea grew out of the success of large [transformers](/tools/transformers). Google's RT-1 (Robotics Transformer, late 2022) showed that a transformer trained on a large set of real robot demonstrations could follow many language instructions. In July 2023 Google DeepMind introduced RT-2, which named the category: it fine-tuned large vision-language models (PaLI-X and PaLM-E) on both web data and robot data, representing each action as text tokens. RT-2 showed better generalisation to new objects and instructions than RT-1 and could follow commands that required web knowledge.

Data was the next bottleneck. In October 2023 the Open X-Embodiment collaboration, involving 21 institutions, pooled more than one million robot episodes across 22 robot types and showed that training on mixed robots could help each one. In June 2024 a team led from Stanford released OpenVLA, a 7-billion-parameter open model that combined a Llama 2 language model with DINOv2 and SigLIP visual features, trained on about 970,000 demonstrations. Its authors reported that it beat the 55-billion-parameter RT-2-X by 16.5 percentage points in absolute success rate across 29 tasks, and that it could be fine-tuned with [LoRA](/tools/lora) on consumer GPUs. Octo, from UC Berkeley and collaborators, offered small open diffusion-based policies in the same period.

Physical Intelligence introduced pi0 in late 2024: a 3.3-billion-parameter model pairing a PaliGemma vision-language backbone with a 300-million-parameter "action expert" trained by flow matching, producing chunks of 50 actions and controlling robots at up to 50 Hz. It was trained on more than 10,000 hours of robot data and demonstrated tasks such as folding laundry and clearing tables. Its code and weights were released as the openpi project in early 2025, followed by pi0.5 (April 2025), which focused on generalising to homes the robot had never seen. During 2025 Google DeepMind launched Gemini Robotics, Figure described Helix, NVIDIA released GR00T N1 for humanoids, and Hugging Face released SmolVLA, a small open model trained on community data. At the time of writing, Google DeepMind's model page lists a second generation, Gemini Robotics 2, and NVIDIA's GR00T line has continued with further versions.

## How it works

### Step 1: encode the scene and the instruction

Camera images, often from a fixed camera and one on the wrist, pass through the vision encoder to become a sequence of image tokens. The instruction is tokenised as text. Robot state, such as joint positions, is projected into the same space. The backbone processes all of these together with [attention](/tools/attention-mechanism), so the instruction can direct which parts of the image matter.

### Step 2: produce actions

With discrete action tokens, each action dimension is normalised and split into bins. With continuous heads, the model learns to turn random noise into a chunk of actions. The key formulas:

```text
Discretisation:   b_i = round( (a_i − a_min,i) / (a_max,i − a_min,i) · (K − 1) )

Policy:           A_t = (a_t, a_{t+1}, …, a_{t+H−1}) ~ π_θ( · | o_t, ℓ )

Flow matching:    A^τ = τ · A + (1 − τ) · ε,      ε ~ N(0, I),  τ ∈ [0, 1]
                  L(θ) = E ‖ v_θ(A^τ, o_t, ℓ, τ) − (A − ε) ‖²
```

In the first line, a_i is one action dimension (for example, the x movement of the gripper), a_min,i and a_max,i are its range in the training data, K is the number of bins (RT-2 used 256), and b_i is the resulting token. In the second, A_t is a chunk of H future actions, o_t the current observation, ℓ the language instruction and π_θ the policy with weights θ. In the flow-matching lines, A is a real action chunk from the data, ε is random noise, τ is a mixing level, and v_θ is the network's predicted direction from noise towards data; this is one common convention for writing the loss. At run time the model starts from noise and follows v_θ in a few steps to produce a smooth action chunk.

### Step 3: execute and repeat

The robot executes the first part of the chunk, captures new images and asks the model again. This closed loop lets the robot correct small errors. Chunking matters because a large model may only run a few times per second, while smooth control needs commands at tens of hertz.

### Step 4: training

Training usually has stages. The backbone is pre-trained on web-scale images and text. Then the whole model is trained on a broad mixture of robot data, often including data from many different robots, sometimes co-trained with web tasks so it does not forget language skills. Finally it is fine-tuned on the target robot and tasks, often with a few dozen to a few hundred demonstrations. This mirrors the pre-train then [fine-tune](/tools/fine-tuning) pattern of language models and is a form of [transfer learning](/tools/transfer-learning).

## Key concepts and terms

| Term | Meaning |
|---|---|
| Embodiment | A specific robot body: its arms, grippers, sensors and joint layout |
| Cross-embodiment training | Training one model on data from several different robots |
| Teleoperation | A human controlling a robot remotely to record demonstrations |
| Behaviour cloning | Learning a policy by imitating recorded demonstrations |
| Action token | An action value turned into a discrete symbol so a language model can output it |
| Action chunk | A short sequence of future actions predicted at once |
| Action expert | A separate network module dedicated to producing continuous actions |
| Flow matching | A generative training method that learns to move noise towards data along a learned direction |
| Degrees of freedom | The number of independent ways a robot can move |
| Control frequency | How many action commands are sent per second |
| Co-training | Mixing robot data with web vision-language data during training |

## A worked example

This scenario is illustrative and does not describe a real company.

A research lab has a two-arm tabletop robot and wants it to sort groceries into a fridge and a cupboard from spoken instructions.

1. **Pick a base model.** They choose an open VLA checkpoint with a vision-language backbone and a continuous action head, already pre-trained on a large multi-robot mixture.
2. **Collect demonstrations.** Two operators teleoperate the robot for about 40 hours, recording roughly 2,000 episodes across 60 grocery items. Each step stores two camera images, the 14 joint positions, the 14-dimensional action and the instruction, for example "put the yoghurt in the fridge".
3. **Fine-tune.** They fine-tune on this data for a day on a few GPUs, holding out 10 items never seen in training.
4. **Evaluate.** On 100 trials with seen items, the robot succeeds 82 times. On 50 trials with the unseen items it succeeds 31 times. Errors cluster on shiny packaging, which confuses depth judgement, and on instructions that use brand names.
5. **Improve.** They add 300 episodes with reflective items and a few varied phrasings of each instruction. Unseen-item success rises to 38 out of 50, while seen-item success stays flat.

The lab reports all numbers with the number of trials, describes the test objects, and records failure videos. That makes the results comparable over time, even though they are not comparable with other labs' setups.

## Strengths and limitations

**Strengths:**

- Language interface: one model can follow many tasks described in ordinary words.
- Transfer of web knowledge: recognising new objects and concepts without robot data for each one.
- One policy for many tasks and, with cross-embodiment training, many robots.
- Efficient fine-tuning: a pre-trained VLA can adapt to a new task with relatively few demonstrations.
- Open models and datasets now exist, so labs can build on shared checkpoints.

**Limitations:**

- Robot data is scarce and expensive compared with web text and images.
- Generalisation is uneven: new objects often work, new skills or new robot bodies much less so.
- Large backbones are slow, which forces action chunking or dual-system designs to reach high control rates.
- Discrete action tokens can limit precision; continuous heads add complexity.
- Long multi-step tasks remain hard, and errors in early steps cascade.
- Safety guarantees are weak: the model has no built-in understanding of force limits or human proximity.
- Results are hard to compare across labs because hardware, objects and scoring differ.

## How it compares with alternatives

| Approach | How actions are chosen | Strengths | Weaknesses |
|---|---|---|---|
| VLA model | One learned model maps images and instruction to actions | General, language-driven, reuses web knowledge | Data-hungry, hard to verify |
| Classical planning and control | Perception, then explicit motion planning and controllers | Predictable, precise, well understood | Brittle with clutter and new objects, heavy engineering |
| Task-specific imitation policy, such as a diffusion policy | Learns one task from demonstrations without a language backbone | Smooth, accurate on its task, small | No language interface, poor transfer to new tasks |
| [Reinforcement learning](/tools/reinforcement-learning) in simulation | Learns by trial and error against a reward | Can discover skills beyond demonstrations | Needs good simulators and rewards, sim-to-real gap |
| Language model planner with skill library | A language model chooses among pre-built skills | Easy to inspect, reuses tested skills | Limited to the skills available, extra latency |

Many deployed systems mix these: a language model or VLA for high-level understanding, classical controllers for safety-critical low-level motion, and [human oversight](/tools/human-in-the-loop) for exceptions.

## Practical guidance and best practices

1. Start from an open pre-trained checkpoint rather than training from scratch; the data needed to pre-train a VLA is out of reach for most teams.
2. Match the action space carefully. Check that the target robot's joints, units, frame conventions and control rate fit what the base model expects, and normalise actions consistently.
3. Collect consistent, high-quality demonstrations. Fewer clean, varied demonstrations usually beat many sloppy ones; vary objects, positions, lighting and phrasing.
4. Hold out objects, scenes and instructions for testing so that you measure generalisation, not memory.
5. Evaluate on the real robot with enough trials to be meaningful, report the number of trials, and keep failure videos. Simulation benchmarks such as LIBERO are useful for fast iteration but do not replace real tests.
6. Wrap the model in a safety layer: joint, speed and force limits, workspace boundaries, emergency stop, and checks that keep the robot from acting near people unless designed for it. See [guardrails](/tools/guardrails) for the general pattern.
7. Use parameter-efficient methods such as [LoRA](/tools/lora) or [PEFT](/tools/peft) to fine-tune on limited hardware, and quantise for deployment only after checking that success rates hold.
8. Log everything in deployment: images, instructions, actions and outcomes. This becomes the next round of training data and the evidence for any incident review.

## Common mistakes

- Reporting success rates from a handful of trials, or on the same objects used in training.
- Mixing action conventions, such as absolute and relative positions or different rotation formats, across datasets.
- Assuming a model fine-tuned on one robot will work on another without new data.
- Trusting language understanding to imply physical competence: knowing what "gently" means does not make the grip gentle.
- Running a large model at too low a frequency without action chunking, which produces jerky, unstable motion.
- Removing classical safety limits because the learned policy "looks careful".

## Frequently asked questions

### How is a VLA different from a vision-language model?

A vision-language model takes images and text and produces text, such as a caption or an answer. A VLA takes the same inputs and produces robot actions. Most VLAs start from a vision-language model and add an action output, then train on robot demonstrations so the language understanding connects to movement.

### Where does the training data come from?

Mostly from humans teleoperating robots and recording what they see and do. Large shared collections such as Open X-Embodiment pool demonstrations from many labs and robot types. Teams add simulation data, videos of humans doing tasks, and web image-text data used in co-training. Collecting high-quality robot data remains the main bottleneck.

### Can a VLA control any robot?

Not reliably out of the box. Cross-embodiment training helps a model share knowledge between robots, but a new body with different joints, cameras and dynamics usually needs fine-tuning on its own demonstrations. Some recent model families claim support for a wide range of bodies, from arms to humanoids; such claims should be checked on your hardware.

### Why do some VLAs use flow matching or diffusion instead of action tokens?

Robot motion is continuous and often needs to be smooth and fast. Turning each action into one of a fixed number of bins can lose precision and makes long, high-frequency action sequences expensive to generate token by token. A continuous head trained with flow matching or diffusion can produce a whole chunk of smooth actions in a few steps, which suits dexterous tasks.

### How are VLAs evaluated?

By running tasks and counting successes. Simulation benchmarks such as LIBERO and SIMPLER give quick, repeatable comparisons. Real-robot evaluations matter more but are expensive and hard to compare, because each lab uses different hardware, objects and scoring rules. Good reports state the number of trials, separate seen from unseen conditions, and describe failures.

### Are VLAs safe to use around people?

Not on their own. A VLA has no built-in guarantee about force, speed or proximity to people, and it can misread an instruction or a scene. Safe deployments wrap the model in conventional safety engineering: limits enforced by the controller, protective stops, restricted workspaces, risk assessments under the relevant machinery and robot safety standards, and human supervision.

## Glossary

- VLA: vision-language-action model, mapping images and instructions to robot actions.
- Vision-language model: a model that processes images and text together and outputs text.
- End effector: the tool at the end of a robot arm, such as a gripper.
- Proprioception: the robot's sense of its own joint positions and forces.
- Behaviour cloning: supervised learning that imitates demonstrated actions.
- Action chunking: predicting several future actions at once and executing them before re-planning.
- Dual-system architecture: a slow reasoning model guiding a fast control model.
- Open X-Embodiment: a large pooled dataset of robot demonstrations from many labs and robot types.
- LIBERO: a simulation benchmark of language-conditioned manipulation tasks.

## Summary

Vision-language-action models combine a vision-language backbone with an action output so that a robot can see a scene, read an instruction and move. RT-2 named the idea in 2023 by writing actions as text tokens; OpenVLA made a strong version open; pi0 introduced a flow-matching action expert for smooth, high-frequency control; and, at the time of writing, Google DeepMind, NVIDIA, Physical Intelligence and open communities continue to release new generations. VLAs transfer web knowledge to new objects and follow many tasks with one model, but they depend on scarce robot data, generalise unevenly to new skills and bodies, and offer no built-in safety guarantees. Start from open checkpoints, collect clean and varied demonstrations, test on held-out objects with enough real trials, and keep classical safety limits and human oversight in place.