Speculative decoding
Speculative decoding lets a small draft model propose several tokens that a large model verifies at once, speeding up generation while keeping the same output distribution.
Overview
Speculative decoding is a way to make a large language model generate text faster without changing what it would produce. Normally a large model writes one token per forward pass, and each pass is slow because the model must read all its weights and its KV cache from memory. Speculative decoding adds a cheap guesser, called the draft. The draft quickly proposes several next tokens. The large model, called the target, then checks all of them in a single pass. Tokens it agrees with are kept, and the first one it disagrees with is replaced.
The trick works because checking several tokens at once costs about the same as generating one. When the draft guesses well, the target accepts several tokens per pass and the whole process runs faster. A careful acceptance rule guarantees that the final text follows exactly the same probability distribution as the target model on its own. In other words, you get the large model's quality at a lower latency.
Think of a senior lawyer and a junior assistant drafting a letter. The assistant types the next few sentences quickly. The lawyer reads them in one go, keeps the parts that are right, and corrects the first sentence that is wrong. The letter reads exactly as the lawyer would have written it, but it gets finished sooner because the lawyer mostly reads rather than writes.
What it is
| Part | Role |
|---|---|
| Target model | The large, accurate model whose output you want |
| Draft mechanism | A cheap way to propose the next few tokens |
| Draft length | How many tokens are proposed per round, often called k |
| Verification pass | One forward pass of the target over all proposed tokens |
| Acceptance rule | A probabilistic test that decides which draft tokens to keep |
| Correction token | The token sampled by the target where the draft was rejected |
| Acceptance rate | The share of draft tokens the target keeps |
The draft can take several forms: a separate small model from the same family, extra prediction heads on the target itself, a light layer that reads the target's hidden features, or even simple text lookup from the prompt.
Where it came from
Speculative decoding was proposed independently by two groups at about the same time. Yaniv Leviathan, Matan Kalman and Yossi Matias of Google Research posted "Fast Inference from Transformers via Speculative Decoding" in November 2022; it was presented at ICML in 2023. A DeepMind team led by Charlie Chen published "Accelerating Large Language Model Decoding with Speculative Sampling" in February 2023. Both used a rejection-sampling rule that keeps the target's output distribution unchanged, and both reported roughly two to three times faster generation in their experiments.
Later work focused on better drafts. Medusa, published in 2024 by Tianle Cai and colleagues, added several extra decoding heads to the target so it could draft for itself without a separate model. EAGLE, also from 2024, drafts by predicting the target's internal features rather than tokens, and follow-up versions added dynamic draft trees. Other variants include prompt lookup, which copies likely continuations from the input, and tree-based verification, which checks several candidate branches at once.
The technique is now built into major serving systems, including vLLM, SGLang and NVIDIA's TensorRT-LLM, and into the Hugging Face Transformers library, where it is called assisted generation.
How it works
Step 1: Draft a few tokens
Given the text so far, the draft proposes k tokens, for example five. Because it is small or simple, this takes little time. It also records the probability it gave each token.
Step 2: Verify in one pass
The target model runs once over the existing text plus the k draft tokens. Because transformers process positions in parallel, this single pass yields the target's own probability for each draft position, plus one more position after the last draft token.
Step 3: Accept or reject each token
Going left to right, each draft token is accepted with probability equal to the smaller of one and the ratio of the target's probability to the draft's probability for that token. If the target likes the token at least as much as the draft did, it is always kept.
Step 4: Correct at the first rejection
At the first rejected position, the method samples a replacement from an adjusted distribution: the target's probabilities minus the draft's, with negatives set to zero and the rest rescaled. All later draft tokens are discarded.
Step 5: Take a bonus token if all pass
If every draft token is accepted, the target's extra prediction at the end gives one more token for free. So each round yields between one and k plus one tokens.
Step 6: Repeat
The process loops until the output is complete. The speed-up depends on the acceptance rate and on how cheap the draft is compared with the target.
Why the output is unchanged
The acceptance and correction rules are designed so that, taken together, each final token has exactly the probability the target alone would have given it. With greedy decoding the rule is simpler: keep draft tokens while they match the target's top choice.
A minimal example with Hugging Face Transformers, where assistant_model is a small model sharing the target's tokenizer:
outputs = model.generate(**inputs, assistant_model=assistant_model, max_new_tokens=200)
Key concepts and terms
| Term | Meaning |
|---|---|
| Draft model | A small model that proposes tokens |
| Target model | The large model that verifies and defines the output |
| Draft length (k) | Number of tokens proposed per round |
| Acceptance rate | Fraction of draft tokens kept |
| Rejection sampling | The probabilistic rule that keeps the output distribution exact |
| Lossless | Producing the same distribution as the target alone |
| Self-speculation | Drafting with parts of the target itself, such as extra heads |
| Draft tree | Several candidate continuations verified together |
| Prompt lookup | Drafting by copying n-grams from the input text |
| Memory-bound | Limited by reading data from memory rather than by arithmetic |
A worked example
An illustrative team serves a 70-billion-parameter model for a coding assistant. One decode step takes about 40 milliseconds, so a 400-token answer takes about 16 seconds.
They add a 1-billion-parameter draft from the same model family. One draft step takes about 4 milliseconds. They set k to 4.
Each round costs four draft steps (16 ms) plus one target verification (about 44 ms, slightly more than a normal step), so about 60 ms.
On their code completions, the average acceptance rate is around 75 per cent per token. With that rate and k of 4, the expected tokens per round work out to roughly 3. That gives about 3 tokens per 60 ms, or 20 ms per token, against 40 ms before.
The 400-token answer now takes about 8 seconds, roughly twice as fast. Outputs are statistically identical to the original. On free-form chat, where the draft guesses less well, acceptance is lower and the gain shrinks to about 1.4 times. The team enables it for code and keeps measuring for chat. The numbers are illustrative.
Strengths and limitations
Strengths:
- Lowers latency without changing output quality, when the standard rule is used.
- Needs no retraining of the target model for draft-model variants.
- Works best on predictable text such as code, structured data and quoted material.
- Supported by mainstream serving engines.
- Can be combined with quantisation and other speed techniques.
Limitations:
- Gains depend on acceptance rate and vary by task.
- The draft needs a compatible tokenizer and extra memory.
- At very large batch sizes, the GPU is already busy and gains can shrink or vanish.
- Adds tuning work: draft choice, draft length and tree shape.
- Self-drafting methods such as Medusa or EAGLE need extra training of heads.
How it compares with alternatives
| Technique | What it speeds up | Changes output? | Notes |
|---|---|---|---|
| Speculative decoding | Per-token latency | No, with standard rule | Needs a good draft |
| Quantisation, for example AWQ or GPTQ | Memory and speed | Slightly | Smaller weights, faster reads |
| Knowledge distillation to a smaller model | Everything | Yes, new model | Lower quality ceiling |
| Batching and paged KV cache | Throughput | No | Helps many users, not single-request latency |
| FlashAttention | Attention cost | No | Complementary kernel-level gain |
Practical guidance and best practices
- Measure baseline latency and throughput on real traffic before enabling it.
- Pick a draft from the same family so tokenizers and style match.
- Start with a draft length of 3 to 5, then tune on your own data.
- Track acceptance rate per task type and switch the feature off where it does not pay.
- Test at your real batch sizes, since gains are largest at low concurrency.
- Consider prompt lookup drafting for tasks that copy from the input, such as editing or summarising code.
- Keep the standard acceptance rule if you need output identical in distribution to the target.
- Watch extra GPU memory used by the draft and its own cache; see cost and latency budgets.
Common mistakes
- Choosing a draft that is too large, so drafting costs eat the gain.
- Using a draft with a different tokenizer, which breaks verification or requires costly conversion.
- Benchmarking only at batch size one and then deploying at high concurrency.
- Assuming a fixed speed-up across tasks.
- Using relaxed acceptance rules for speed without telling users that outputs may differ.
Frequently asked questions
Does speculative decoding reduce quality?
With the standard rejection-sampling rule, no. The output follows the same distribution as the target model alone. Some variants relax the rule for more speed, and those can change outputs.
How much faster is it?
It depends on the task, the draft and the hardware. The original papers reported around two to three times in their settings. In practice gains range from small to several times, so measure on your own workload.
Do I need to train a draft model?
Not always. A smaller model from the same family often works well as is. Methods such as Medusa and EAGLE do need training of extra heads or layers, but this is much cheaper than training the target.
Why does it help less with large batches?
Speculative decoding uses spare compute during memory-bound decoding. When many requests are batched together, the GPU is already doing more arithmetic, so there is less spare capacity to exploit.
Can it be used with reasoning models?
Yes. Long reasoning traces are a good fit, because generation is long and latency matters. Acceptance rates depend on how predictable the trace text is.
Is it the same as caching?
No. Caching reuses past computation. Speculative decoding guesses future tokens and verifies them. They work together.
Glossary
- Forward pass: one run of the model over its input.
- Token: a small unit of text the model reads and writes.
- Greedy decoding: always choosing the most likely next token.
- Sampling: choosing the next token randomly according to its probability.
- Distribution: the set of probabilities over all possible next tokens.
- Latency: time taken to answer one request.
- Throughput: amount of work done per second across all requests.
- Tokenizer: the component that splits text into tokens.
Summary
Speculative decoding speeds up large language models by letting a cheap draft propose several tokens and having the large model verify them in one pass, with a rejection-sampling rule that keeps the output distribution exactly the same. It was introduced independently by Google Research and DeepMind in 2022 and 2023, and extended by self-drafting methods such as Medusa and EAGLE. It is now built into major serving engines. Gains depend on how often the draft is right and on batch size, so it should be tuned and measured on real workloads.
Related
It belongs to LLM concepts in my stack, under Generative AI and LLMs. Others in the same group: