World models

Overview

A world model is a neural network that has learned how some part of the world behaves. Give it the current situation and an action, and it predicts what happens next. Chain those predictions together and the model becomes a simulator that an agent can use to plan ahead, test ideas or practise skills without touching the real environment.

The idea answers a practical problem. Learning by trial and error in the real world is slow, expensive and sometimes dangerous. A robot arm that drops a glass a thousand times while learning to pour is not acceptable, and a self-driving car cannot crash to learn what crashing means. If the agent can first learn a good enough model of how things move and respond, it can make most of its mistakes in imagination. This is the core of model-based reinforcement learning, and it is also why large video generators are now described as "world simulators".

An everyday picture: before you carry a full tray of drinks across a crowded room, you run a quick mental film. You imagine the step around the chair, the child who might dart past, the tilt of the tray. You never rehearsed this exact room, but your internal model of gravity, liquids and people lets you predict trouble and pick a safer path. A world model gives a machine a learned version of that mental film. Like ours, it is only as good as the experience it was built from, and it can be confidently wrong.

What it is

There are two broad families. Latent dynamics models predict in a compressed internal space and only decode to pixels when needed. They are small, fast and built to support decision-making, as in the Dreamer line of agents. Generative world models predict observations directly, usually as video, and are built to look realistic and respond to controls. Genie 3 and the video models used for driving simulation belong here. A third, related view argues that a world model should predict abstract representations rather than pixels at all; joint-embedding predictive architectures (JEPA) take this route.

What unites them is the conditional prediction: given what has happened and what the agent does, what comes next? A plain video generator that cannot be steered by actions is not a full world model, although it may contain much of the same knowledge.

Where it came from

The idea that an agent should carry an internal model of its surroundings is old in psychology and control engineering. In machine learning, Jürgen Schmidhuber's work around 1990 and 1991 proposed recurrent networks that learn to predict the consequences of actions and a controller that learns by using those predictions. Around the same time Richard Sutton's Dyna architecture (1990) mixed real experience with simulated experience drawn from a learned model, a pattern still used today.

The modern revival is usually dated to David Ha and Jürgen Schmidhuber's 2018 paper "World Models", presented at NeurIPS 2018 as "Recurrent World Models Facilitate Policy Evolution". Their agent had three parts: a VAE that compressed each frame, an LSTM-based mixture density network that predicted the next compressed frame, and a tiny linear controller trained with an evolution strategy. In a car-racing task and a Doom-based dodging game, the controller could be trained entirely inside the model's "dream" and then transferred to the real game. The paper also documented a lasting lesson: a controller can learn to exploit errors in an imperfect model, and adding randomness to the dream made the learned behaviour transfer better.

Danijar Hafner and colleagues then built the latent-dynamics line. PlaNet (2019) planned in a learned latent space. Dreamer (2020) trained an actor-critic entirely on imagined trajectories. DreamerV2 (2021) reached strong Atari results with discrete latents. DreamerV3, posted in 2023 and published in Nature in April 2025, used one fixed set of hyperparameters across more than 150 tasks and was the first algorithm to collect diamonds in Minecraft from scratch, without human data or curricula. In parallel, MuZero (Nature, 2020) learned a model that predicted only what planning needed, mastering Go, chess, shogi and Atari without being told the rules.

The generative branch grew out of video modelling. In 2024 Google DeepMind's Genie learned playable 2D environments from unlabelled internet video, and GameNGen showed a diffusion model running the game DOOM interactively at over 20 frames per second on a single TPU. Genie 3, announced in August 2025, generates explorable 3D worlds at 720p and 24 frames per second that stay consistent for a few minutes, as a limited research preview. Meta released V-JEPA 2 in June 2025, a representation-predicting video model used for zero-shot robot planning.

At the time of writing, the field is moving fast. In February 2026 Waymo described the Waymo World Model, built on Genie 3 and adapted to produce camera and lidar data for simulating rare driving events. In May 2026 NVIDIA released Cosmos 3, an open model family that combines vision reasoning, world generation and action prediction for robotics and autonomous vehicles.

How it works

Step 1: collect experience

The model learns from sequences of observations, actions and, if available, rewards. For a robot this means camera frames plus motor commands; for a game, screen frames plus key presses. Generative models trained on internet video often lack action labels, so some, like the original Genie, infer a small set of latent actions from how frames change.

Step 2: learn a compact state

An encoder maps each observation to a latent state. The recurrent state-space model (RSSM) used by PlaNet and Dreamer splits this state into a deterministic memory, which carries information across time, and a stochastic part, which captures what cannot be predicted with certainty. The key formulas, in one common notation:

h_t  = f_θ(h_{t−1}, z_{t−1}, a_{t−1})          deterministic memory
z_t  ~ q_θ(z_t | h_t, x_t)                      posterior: state after seeing the frame
ẑ_t  ~ p_θ(ẑ_t | h_t)                           prior: state predicted without the frame
x̂_t ~ p_θ(x_t | h_t, z_t),  r̂_t ~ p_θ(r_t | h_t, z_t)

L(θ) = Σ_t [ −log p_θ(x_t | h_t, z_t) − log p_θ(r_t | h_t, z_t)
             + β · KL( q_θ(z_t | h_t, x_t) ‖ p_θ(ẑ_t | h_t) ) ]

Here x_t is the observation at time t, a_t the action, r_t the reward, h_t the recurrent memory, z_t the stochastic latent, θ all learned weights, KL the Kullback-Leibler divergence from information theory, and β a weight that balances reconstruction against the KL term. The KL term trains the prior, which only sees the past, to guess what the posterior learns after seeing the frame. Once the prior is good, the model can roll forward without new observations: that is imagination.

Step 3: predict forward

To imagine, start from a real latent state, pick an action, and let the dynamics model produce the next state, repeatedly. Latent models do this cheaply, often thousands of steps in parallel on a GPU. Video world models do the equivalent by generating frames autoregressively, each new frame conditioned on recent frames and the latest action, usually with a diffusion model or an autoregressive transformer.

Step 4: use the predictions

There are three main ways to use a world model:

• Planning: search over action sequences in imagination and execute the best first action, then re-plan. PlaNet and MuZero work this way.
• Learning in imagination: train a policy and value function on imagined rollouts, as Dreamer does, so most learning never touches the real environment.
• Data generation and testing: produce synthetic scenarios, such as rare weather or unusual road users, to train or stress-test other systems.

Step 5: keep it honest

Errors compound over long rollouts, so practical systems keep imagined horizons short, re-anchor on real observations often, and keep collecting fresh real data as the policy changes. The policy will always find situations the model has not seen, so real-world checks remain essential.

Key concepts and terms

A worked example

This scenario is illustrative and does not describe a real company.

A small warehouse team wants a robot arm to push boxes into a target zone on a table. Real trial and error is slow: each attempt takes about 20 seconds including reset, so 10,000 attempts would take more than two days of continuous operation, with wear on the hardware.

1. Collect data. The team records 300 episodes of 50 steps each, about 15,000 transitions, using a simple scripted pusher plus random actions. Each step stores a 64×64 camera image, the arm's two-dimensional push command and a reward equal to minus the box's distance from the target.
2. Train a latent model. They train a small RSSM-style model. After training, one-step predictions of box position are within about a centimetre on held-out episodes, but 30-step predictions drift by several centimetres.
3. Learn in imagination. They roll out 15-step imagined trajectories from thousands of real starting states and train an actor-critic on them. Because the horizon is short, compounding error stays manageable.
4. Test on the robot. The first real trial succeeds in about 6 out of 10 attempts. Failures cluster when two boxes touch, a case barely present in the data.
5. Close the loop. They run the new policy for another 100 real episodes, add the data, retrain the model and policy. Success rises to about 8 out of 10.

Two lessons stand out. The model made learning far cheaper, because most of the policy's experience was imagined. And the model's blind spot, touching boxes, became the policy's blind spot until new real data covered it.

Strengths and limitations

Strengths:

• Sample efficiency: an agent can learn from far fewer real interactions by practising in imagination.
• Planning: the agent can look ahead and compare possible futures before acting.
• Safety in training: risky situations can be explored in simulation first.
• Reuse: one learned model can serve many tasks or reward functions in the same environment.
• Synthetic scenarios: generative world models can produce rare or hard-to-collect situations for training and testing.
• Inspectability: decoding imagined rollouts lets engineers see what the model expects.

Limitations:

• Compounding error makes long-horizon predictions drift.
• Policies exploit model errors, which can produce behaviour that works only in the dream.
• Realistic video is not the same as correct physics; a model can look right while violating cause and effect.
• Large video world models are expensive to train and to run in real time.
• Coverage is limited by data: situations absent from training are predicted poorly.
• Evaluation is unsettled, and good scores on one measure do not guarantee usefulness for control.

How it compares with alternatives

In practice these combine. Robotics teams often pre-train in a hand-built simulator, use a learned model to close the gap with real data, and fine-tune with model-free methods.

Practical guidance and best practices

1. Decide what the model is for. A model used for planning needs accurate action effects and rewards; a model used for visual scenario generation needs realism and controllability. Do not optimise one and assume the other.
2. Start with latent dynamics for control problems. They are cheaper to train and roll out than pixel-level video models and are well understood.
3. Keep imagination horizons short and re-anchor on real observations often to limit compounding error.
4. Inject stochasticity and use ensembles. Disagreement between several models is a useful signal of uncertainty and helps prevent model exploitation.
5. Collect data iteratively. Each new policy visits new states, so retrain the model on data from the current policy, not only from the initial dataset.
6. Measure what matters downstream: task success on the real system, not only prediction loss or video quality.
7. Test rare cases on purpose: object interactions, occlusion, lighting changes and unusual agents, since these are where models fail.
8. Keep a real-world safety layer. Imagined success is a hypothesis; confirm it with controlled real tests, human oversight and hard limits on actuators.

Common mistakes

• Judging a world model by how good its video looks rather than by whether actions have the right effects.
• Rolling out for hundreds of steps and trusting the result without checking drift against reality.
• Training once on a fixed dataset and never refreshing it as the policy improves.
• Ignoring model exploitation: a policy that scores highly in imagination but fails in the real environment.
• Treating a text-only language model as a world model of physical dynamics without testing it on physical prediction.
• Comparing systems across papers whose benchmarks, horizons and data budgets differ.

Frequently asked questions

Is a large language model a world model?

Partly, and the question is debated. Language models capture a lot of knowledge about how the world works, and some research suggests they build internal representations of the situations they read about. But they are trained to predict text, not the consequences of physical actions, and they often fail at consistent physical prediction. Researchers such as Yann LeCun argue that text-only training cannot give a full model of the physical world. A practical rule is to treat any model as a world model only for the kinds of prediction you have actually tested.

What is the difference between a world model and a simulator?

A traditional simulator is written by engineers from equations and assets. A world model is learned from data. Learned models can capture effects that are hard to write down, such as cloth, crowds or camera noise, but they are less precise and harder to verify. Many teams use both: a hand-built simulator for structure and a learned model to add realism or adapt to real data.

Why do latent models not predict pixels directly?

Pixels contain a lot of detail that does not matter for decisions, such as background texture. Predicting in a compact latent space is cheaper and lets the model focus on what changes the outcome. The decoder is still useful for training and for humans to inspect what the model imagines, but planning and policy learning happen in the latent space.

How are world models evaluated?

There is no single standard. Common measures include prediction error on held-out data, visual quality and consistency for generative models, physical-reasoning benchmarks such as IntPhys 2, and, most importantly, downstream task success when an agent uses the model. Sources agree that these measures do not always line up: a model can predict video well and still guide a robot poorly.

Do world models help self-driving cars?

They are used mainly for simulation and testing. Generative driving models can create rare and safety-critical situations, such as unusual road users or extreme weather, so driving software can be exercised on cases that are hard to record. At the time of writing, Waymo has described using a Genie 3-based model in this way. Real-world validation and conventional safety engineering remain necessary.

How do world models relate to vision-language-action models?

They are complementary. A vision-language-action model maps observations and instructions directly to robot actions. A world model predicts what those actions will do. Some recent systems combine the two, using world-model pre-training or imagined rollouts to improve action models, and NVIDIA's Cosmos 3, released in 2026, bundles world generation and action prediction in one model family.

Glossary

• World model: a learned model that predicts how an environment changes in response to actions.
• RSSM: recurrent state-space model, a latent dynamics model with deterministic and stochastic state parts.
• Dreamer: a family of agents that learn behaviour by imagining trajectories in a learned latent model.
• MuZero: a planning agent that learns a model predicting only what search needs: value, policy and reward.
• Imagination: rolling the model forward without real observations to produce hypothetical futures.
• Latent action: an inferred action label learned from how video frames change when no real actions are recorded.
• Sim-to-real transfer: moving a policy trained in simulation to the real world.
• Ensemble disagreement: the spread between several models' predictions, used as an uncertainty signal.
• Autoregressive generation: producing each new step conditioned on the previous ones.

Summary

A world model is a learned predictor of how an environment changes when an agent acts. Latent dynamics models such as the Dreamer family compress observations into small states and let agents plan or learn in imagination, which makes learning far cheaper in data. Generative world models such as Genie 3 produce controllable video worlds and, at the time of writing, are used to create training and test scenarios for robots and autonomous vehicles. The idea traces back to Schmidhuber's work around 1990 and Sutton's Dyna, and returned to prominence with Ha and Schmidhuber's 2018 paper. The main risks are compounding error, policies that exploit model mistakes, and realistic-looking predictions that break physics. Use world models with short horizons, fresh real data, uncertainty estimates and real-world validation, and judge them by what they help an agent do rather than by how convincing they look.

Related

It belongs to Frontier topics in my stack, under Deep learning. Others in the same group: