# Diffusion language models

Diffusion language models write text by starting from a fully masked draft and refining many tokens at once, trading strict left-to-right decoding for speed and editing.

## Overview

Almost every well-known large language model is autoregressive. It writes one token, appends it to the input, and predicts the next, always moving left to right. This is simple and works extremely well, but it has a built-in speed limit: a 1,000-token answer needs about 1,000 sequential steps, and an early mistake cannot be revisited.

Diffusion language models take a different route. Borrowing the core idea behind image generators such as [Stable Diffusion](/tools/stable-diffusion), they begin with a corrupted version of the output and repeatedly clean it up. For text, the most successful form of corruption is masking: the answer starts as a row of blank placeholders, and in each step the model fills in some of them while looking at the whole sequence at once. After a number of refinement steps, every position holds a real token. Because each step can commit many tokens in parallel, the number of sequential steps can be much smaller than the length of the answer.

An everyday analogy: an autoregressive model writes like someone typing a letter from the first word to the last without ever going back. A diffusion model writes like someone filling in a crossword. They pencil in the answers they are surest of first, anywhere on the grid, and use those letters to help with the harder clues. By the end, the grid is complete, but it was not filled in order.

## What it is

| Component | Role |
|---|---|
| Forward (noising) process | Gradually corrupts clean text, usually by replacing tokens with a mask token |
| Mask token | A special placeholder meaning "unknown token here" |
| Denoiser | A [transformer](/tools/transformers) without a causal mask that predicts the original tokens at masked positions |
| Noise level or time step | How much of the sequence is masked, from none to all |
| Reverse (sampling) process | Starts fully masked and unmasks tokens over a set number of steps |
| Remasking or unmasking strategy | Rules for which predictions to keep at each step, for example the most confident ones |
| Block structure (optional) | Splits a long output into blocks generated in order, with diffusion inside each block |

The key architectural difference from an autoregressive model is the attention pattern. An autoregressive transformer uses a causal mask, so each position sees only earlier positions. A diffusion denoiser uses full bidirectional [attention](/tools/attention-mechanism), like [BERT](/tools/bert), so every position can see both its left and right context. That is what lets it fill blanks anywhere and in any order.

There are two families. Continuous diffusion models map tokens into vectors, add Gaussian noise, and learn to remove it, as in image diffusion. Discrete diffusion models corrupt the tokens themselves. Discrete masked diffusion has become the dominant approach for language because it fits the discrete nature of text and connects naturally to masked language modelling.

## Where it came from

Diffusion models first became prominent for images, with denoising diffusion probabilistic models ([DDPM](/tools/ddpm)) in 2020 and their faster samplers such as [DDIM](/tools/ddim). Applying the idea to text required handling discrete tokens. In 2021, Jacob Austin and colleagues at Google introduced D3PM, "Structured Denoising Diffusion Models in Discrete State-Spaces", at NeurIPS. It defined several ways to corrupt discrete data, including an absorbing state where tokens turn into a mask, which is the ancestor of today's masked diffusion language models.

In 2022, Xiang Lisa Li and colleagues at Stanford proposed Diffusion-LM, a continuous approach that diffused word embeddings and was aimed at controllable text generation. Discrete methods then improved steadily. Aaron Lou, Chenlin Meng and Stefano Ermon introduced score entropy discrete diffusion (SEDD) in "Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution", which won a best paper award at ICML 2024. Subham Sahoo and colleagues published "Simple and Effective Masked Diffusion Language Models" (MDLM) at NeurIPS 2024, showing that a cleanly formulated masked diffusion objective closes much of the quality gap with autoregressive models. In 2025, Marianne Arriola and colleagues presented block diffusion as an oral paper at ICLR 2025, interpolating between autoregressive and diffusion generation.

Scale arrived in 2025. In February, Shen Nie and colleagues from Renmin University of China and Ant Group released LLaDA, an 8-billion-parameter masked diffusion model trained from scratch on 2.3 trillion tokens, which they reported as competitive with LLaMA3 8B on in-context learning. Also in February 2025, Inception Labs launched Mercury, presented as the first commercial-scale diffusion language model; its June 2025 technical report described Mercury Coder Mini and Small reaching 1,109 and 737 tokens per second on NVIDIA H100 GPUs in independent measurements by Artificial Analysis. In April 2025, the University of Hong Kong and Huawei Noah's Ark Lab released Dream 7B, initialised from the autoregressive Qwen2.5 7B weights. On 20 May 2025, Google DeepMind announced Gemini Diffusion as an experimental research model with a waitlisted demo, saying it generated content significantly faster than its fastest model at the time.

In December 2025, the LLaDA2.0 report from Ant Group and collaborators described converting pretrained autoregressive models into diffusion models, producing mixture-of-experts models of 16 billion and 100 billion total parameters. In February 2026, Inception announced Mercury 2, described as the first reasoning diffusion language model, with a stated throughput of about 1,000 tokens per second, and the company has announced further versions since. These are vendor claims; independent comparisons are still limited at the time of writing.

## How it works

### Training: learn to fill blanks at every noise level

Training is close to masked language modelling with one important change. For each training sequence, a masking ratio t is drawn at random between 0 and 1, and each token is independently replaced by the mask token with probability t. The model sees the partly masked sequence and is trained with cross-entropy to predict the original token at every masked position. BERT uses a fixed masking ratio; a diffusion model sees every ratio from almost nothing masked to everything masked. That is what allows it to start from a fully blank sequence at generation time.

### The training objective

```text
Forward process:  each token x₀ⁱ becomes [MASK] with probability t,  t ~ U(0, 1)
Loss:             L(θ) = − E_{t, x₀, x_t} [ (1/t) · Σ_{i : x_tⁱ = [MASK]}  log p_θ( x₀ⁱ | x_t ) ]
Sampling:         x_1 = all [MASK]  →  x_{t−Δ}  →  …  →  x_0 = clean text
```

Here x₀ is the clean sequence and x₀ⁱ is its i-th token, t is the masking ratio or noise level, and x_t is the sequence after masking at that level. The sum runs over positions that are masked, p_θ is the model's predicted probability for the original token given the whole partly masked sequence, and θ are the model's weights. The 1/t factor weights low-noise examples more heavily, and the whole expression is an upper bound on the negative log-likelihood, which is why LLaDA's authors describe it as a principled generative objective. In the sampling line, generation starts from a fully masked sequence x_1 and moves in steps of size Δ towards x_0.

### Sampling: unmask a few, keep the best, repeat

Generation starts with the prompt followed by a fixed number of mask tokens for the answer. At each step, the model predicts a token for every masked position at once. A strategy then decides which of those predictions to commit. A common choice is to keep the most confident predictions and leave the rest masked (or remask low-confidence ones) for the next step. After a chosen number of steps, every position is filled.

The number of steps is the main speed and quality dial. If the answer is 256 tokens long and you use 256 steps, you commit roughly one token per step and gain little speed. With 64 steps you commit about four per step, and the process is much faster but each step must guess more tokens jointly, which tends to lower quality.

### Block diffusion and caching

Pure diffusion has two practical problems. It needs the output length fixed in advance, and because attention is bidirectional, the standard [KV cache](/tools/kv-cache) that makes autoregressive decoding efficient does not apply directly. Block diffusion addresses both: it generates the output in blocks from left to right, using autoregressive-style caching across blocks, while running diffusion inside each block. This allows variable-length output and reuse of computed context, at the cost of some of the any-order flexibility.

## Key concepts and terms

| Term | Meaning |
|---|---|
| Autoregressive model | A model that generates one token at a time, each conditioned on earlier ones |
| Discrete diffusion | Diffusion that corrupts tokens directly rather than continuous vectors |
| Masked (absorbing) diffusion | Discrete diffusion where corruption turns tokens into a mask token |
| Denoiser | The network that predicts clean tokens from a corrupted sequence |
| Bidirectional attention | Attention in which every position sees both left and right context |
| Sampling steps | The number of refinement passes used to produce an output |
| Parallel decoding | Committing more than one token per forward pass |
| Remasking | Returning low-confidence predictions to the mask state for later steps |
| Block diffusion | Left-to-right generation of blocks with diffusion inside each block |
| AR-to-diffusion conversion | Initialising a diffusion model from pretrained autoregressive weights |
| Infilling | Generating text in the middle of existing text, given both sides |

## A worked example

Consider an illustrative code assistant that must fill in the body of a function between a known signature and a known return statement. The gap is 120 tokens.

An autoregressive model would need about 120 sequential decoding steps, and it would normally only see the text before the gap unless it was specially trained for fill-in-the-middle. A masked diffusion model places 120 mask tokens in the gap. It sees the signature on the left and the return statement on the right from the very first step.

Suppose we choose 30 sampling steps, so about four tokens are committed per step. In step one, the model predicts all 120 positions. It is very confident about a few: the opening of a loop that must match the return type, and the variable name that appears in the return statement. It commits the four most confident tokens and leaves the rest masked. In step two, with those anchors fixed, the predictions for neighbouring positions sharpen, and four more are committed. After 30 steps, the gap is filled.

Now the team experiments. With 120 steps, quality is slightly higher but the speed advantage disappears. With 15 steps, it is roughly twice as fast again, but tests start failing more often because neighbouring tokens were chosen independently and occasionally clash, for example an opening bracket without its closing partner. They settle on a step count that passes their test suite at an acceptable rate, and they treat the step count as a tunable setting per task, not a fixed constant. The numbers here are illustrative; real speed depends heavily on hardware, model size and implementation.

## Strengths and limitations

**Strengths:**

- Parallel decoding can generate many tokens per forward pass, which vendors report as much higher throughput than comparable autoregressive models.
- Bidirectional context makes infilling and editing natural, since the model sees both sides of a gap.
- Later steps can in principle revise earlier choices through remasking, unlike strictly left-to-right decoding.
- Speed and quality can be traded at inference time by changing the number of steps.
- LLaDA's authors reported more consistent behaviour on reversal tasks, such as completing a poem line from the following line.

**Limitations:**

- Quality typically drops as the number of tokens committed per step rises, because tokens chosen in the same step are predicted independently.
- Plain diffusion needs a fixed output length; block diffusion and other tricks are needed for open-ended generation.
- Standard KV caching does not directly apply to bidirectional attention, which offsets some of the speed gain.
- The surrounding ecosystem of serving engines, tooling and fine-tuning recipes is far smaller than for autoregressive models.
- At the time of writing, the best-known frontier models are still autoregressive, and independent quality comparisons for diffusion models are limited.
- Training from scratch is costly, which is why conversion from autoregressive models has become popular.

## How it compares with alternatives

| Approach | How it generates | Main advantage | Main drawback |
|---|---|---|---|
| Diffusion language model | Refines the whole sequence over a few steps | Parallel decoding, natural infilling | Quality and step-count trade-off, younger ecosystem |
| Autoregressive LLM ([GPT-style](/tools/gpt-architecture)) | One token at a time, left to right | Mature, highest-quality frontier models | Sequential steps scale with output length |
| [Speculative decoding](/tools/speculative-decoding) | Draft several tokens, verify with the main model | Same output as the main model, faster | Gains depend on draft acceptance rate |
| Multi-token prediction heads | Predicts several next tokens per step | Faster autoregressive decoding | Still fundamentally left to right |
| Masked language model ([BERT](/tools/bert)) | Fills masks in one pass at a fixed ratio | Strong for understanding tasks | Not designed to generate long text |

Speculative decoding is the closest competitor for speed, because it keeps an autoregressive model's exact output distribution while drafting several tokens at a time. Diffusion models change the generation process itself, which gives them infilling and editing abilities that speculative decoding does not, but also exposes them to the parallel-decoding quality trade-off.

## Practical guidance and best practices

1. Decide what you need first: raw throughput, infilling and editing, or best-possible quality. Diffusion models are strongest on the first two.
2. Treat the number of sampling steps as a key hyperparameter and tune it per task with your own evaluation set.
3. Check how the model handles output length. Know whether it uses fixed-length generation, block diffusion or another mechanism.
4. Measure end-to-end latency on your own hardware and prompts, including prompt processing, rather than relying on headline tokens-per-second figures.
5. Use automatic checks such as tests, parsers or [structured output](/tools/structured-outputs) validators, because fast parallel generation is most useful when you can cheaply catch its occasional inconsistencies.
6. For open-weight models, check the licence and the serving support in your inference stack before committing to a design.
7. Compare against a well-tuned autoregressive baseline with speculative decoding, not against naive decoding.

## Common mistakes

- Assuming fewer steps are free. Cutting steps raises speed but usually lowers quality, sometimes sharply.
- Comparing vendor throughput numbers measured on different hardware, batch sizes and prompt lengths as if they were equivalent.
- Expecting autoregressive serving optimisations, such as standard KV caching, to apply unchanged.
- Treating a diffusion model as a drop-in replacement for an autoregressive chat model without re-running your evaluations.
- Confusing diffusion language models with image diffusion models; they share an idea, not an architecture or a training set.

## Frequently asked questions

### Are diffusion language models the same as image diffusion models?

They share the core idea of learning to reverse a gradual corruption process. Image models usually add continuous Gaussian noise to pixels or latents, while most successful language models use discrete masking of tokens. The networks, data and sampling details differ substantially.

### Why are diffusion language models fast?

An autoregressive model needs one forward pass per generated token. A diffusion model can commit several tokens per forward pass, so a long answer needs fewer sequential passes. Actual speed depends on the number of steps, the model size, caching and the hardware.

### Can they edit or revise text?

Yes, more naturally than autoregressive models. Because the denoiser sees context on both sides, it can fill a gap in the middle of a document. Remasking strategies also let later steps replace earlier low-confidence choices, although how much revision actually happens depends on the sampler.

### Are they as good as the best autoregressive models?

At the time of writing, research models such as LLaDA have been reported as competitive with autoregressive models of similar size on many benchmarks, and commercial providers claim quality comparable to fast autoregressive models. The strongest general-purpose frontier models remain autoregressive, and independent head-to-head evidence is still limited.

### Can I convert an existing autoregressive model into a diffusion model?

Research suggests yes. Dream 7B was initialised from Qwen2.5 7B, and the LLaDA2.0 work describes continued pretraining that converts autoregressive models into diffusion models at up to 100 billion parameters. Conversion is far cheaper than training from scratch but still requires substantial compute.

### Do diffusion language models work with reasoning?

Inception describes Mercury 2 as a reasoning diffusion model, and research groups are exploring reinforcement learning for diffusion models. The field is moving quickly, so check current evidence before relying on reasoning performance for a specific task.

## Glossary

- Diffusion language model (dLLM): a language model that generates by iterative denoising.
- Mask token: the placeholder that marks an unknown position.
- Forward process: the corruption applied to clean text during training.
- Reverse process: the step-by-step denoising used at generation time.
- Denoiser: the network that predicts clean tokens from a corrupted input.
- Sampling steps: the number of refinement passes during generation.
- Parallel decoding: committing more than one token per step.
- Remasking: returning uncertain predictions to the masked state.
- Block diffusion: blockwise left-to-right generation with diffusion inside each block.
- Infilling: generating missing text between two known pieces.

## Summary

Diffusion language models generate text by refining a whole sequence over several steps instead of writing one token at a time. Most use masked diffusion: training teaches a bidirectional transformer to fill blanks at every masking level, and generation starts from a fully masked answer and commits the most confident tokens step by step. This enables parallel decoding, natural infilling and adjustable speed, which is why projects such as LLaDA, Dream, Mercury and Gemini Diffusion have attracted attention. The price is a trade-off between speed and quality as more tokens are committed per step, extra machinery for variable length and caching, and a less mature ecosystem. At the time of writing they are a serious and fast-moving alternative to autoregressive models, best evaluated on your own tasks against a well-tuned autoregressive baseline.