Özer Cem KELAHMETKELAHMET
AI Simulations · Language models

Speculative decoding: a small model drafts, the large model verifies

Large language models generate one token per forward pass, and each pass is limited by how fast the weights can be read from memory. Speculative decoding lets a small, fast model guess the next few tokens and has the large model verify all of them at once. When the guesses are good, you get several tokens for the price of one large pass, with exactly the same output distribution.

Level AdvancedAbout 10 minRead first: Sampling, KV cache1 challenge in the Lab

Interactive simulation. It needs JavaScript; the text below explains what it shows.

What you are looking at

Both models are small probability tables over a 12-word vocabulary, so every number is exact. Draft quality controls how close the draft's distribution q is to the large model's p. The theory panel computes the acceptance rate β, the expected tokens per large-model pass and the speed-up from the formulas. Generate runs the real algorithm: blue tokens were drafted and accepted, orange ones were rejected and resampled by the large model, white ones are the bonus token added when every draft was accepted. Check the output distribution samples the next token 3,000 times to show that speculative sampling follows p, not q.

Things to try

  • Start with the defaults and press Generate: count how many blue tokens appear in a row.
  • Lower Draft quality to 20%: acceptance falls and the speed-up drops towards 1×.
  • With a good draft, raise the draft length from 2 to 8: the curve shows a best k, after which extra drafts are wasted.
  • Make the draft cost 50% of the large model: even good guesses no longer pay.
  • Press Check the output distribution with a poor draft: the speculative column still matches the large model.

How it works in real systems

Production servers such as vLLM, TensorRT-LLM and Hugging Face Transformers support speculative decoding. The draft is either a small model from the same family, which must share the tokenizer, or extra prediction heads on the large model itself (Medusa, EAGLE). The gain is largest at small batch sizes, where decoding is memory-bound; at large batches the GPU is busier and the speed-up shrinks. Because of the rejection rule the method is lossless for sampling, which is why providers can use it without changing model behaviour.

Key formulas

$$\beta=\sum_x \min\big(p(x),q(x)\big)=1-\mathrm{TV}(p,q)$$

The acceptance rate: how much the draft distribution q overlaps the large model's p.

$$\mathbb E[\text{tokens per pass}]=\frac{1-\beta^{k+1}}{1-\beta},\qquad \text{speed-up}=\frac{1-\beta^{k+1}}{(1-\beta)(kc+1)}$$

Leviathan et al. (2023), with c the draft's cost relative to the large model.

$$\text{keep } x\sim q \text{ with prob. } \min\Big(1,\frac{p(x)}{q(x)}\Big),\ \text{else sample } \frac{\max(0,\,p-q)}{\lVert\max(0,\,p-q)\rVert_1}$$

The rejection rule that keeps the output distribution exactly p.

Common mistakes

  • Measuring speed-up at batch size 1 only: at large batch sizes the large model is less memory-bound and the gain shrinks.
  • Choosing a draft model with a different tokenizer, which makes token-level verification impossible.
  • Using a long draft length with a weak draft: rejected tokens are wasted compute.

Real-world numbers

  • Leviathan et al. (2023) reported 2–3× faster T5-XXL inference with identical outputs.
  • Chen et al. (2023) reported 2–2.5× faster decoding for the 70B Chinchilla model.
  • Medusa and EAGLE replace the separate draft model with extra heads on the large model itself.

Further reading

Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

Step 1 · Speculative sampling from scratch

Two toy distributions, the rejection rule, and a check that the output matches the large model.

pip install numpy transformers torch accelerate
import numpy as np
rng = np.random.default_rng(0)
V = 12
p = rng.dirichlet(np.ones(V) * 0.5)          # the large model's next-token distribution
q = 0.6 * p + 0.4 * rng.dirichlet(np.ones(V))  # an imperfect draft

def speculative_token():
    x = rng.choice(V, p=q)                      # the draft proposes
    if rng.random() < min(1, p[x] / q[x]):      # the large model accepts
        return x
    r = np.maximum(p - q, 0)                    # or resamples what q under-weighted
    return rng.choice(V, p=r / r.sum())

samples = np.bincount([speculative_token() for _ in range(100_000)], minlength=V) / 100_000
print('max |speculative - p| =', np.abs(samples - p).max().round(4))
print('acceptance rate beta =', np.minimum(p, q).sum().round(3))

Step 2 · Expected speed-up

Leviathan et al.'s formula for k drafted tokens and a draft that costs c of a large pass.

def speedup(beta, k, c):
    tokens = (1 - beta ** (k + 1)) / (1 - beta)
    return tokens / (k * c + 1)

beta = np.minimum(p, q).sum()
for k in range(1, 9):
    print(k, round(speedup(beta, k, c=0.1), 2))

Step 3 · Assisted generation in Transformers

Pass a small model from the same family as assistant_model. Both models must use the same tokenizer.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch, time

big, small = 'HuggingFaceTB/SmolLM2-1.7B-Instruct', 'HuggingFaceTB/SmolLM2-135M-Instruct'
tok = AutoTokenizer.from_pretrained(big)
model = AutoModelForCausalLM.from_pretrained(big, torch_dtype=torch.bfloat16, device_map='auto')
draft = AutoModelForCausalLM.from_pretrained(small, torch_dtype=torch.bfloat16, device_map='auto')

inputs = tok('Explain speculative decoding in two sentences.', return_tensors='pt').to(model.device)
for name, kw in [('plain', {}), ('speculative', {'assistant_model': draft})]:
    t = time.time()
    out = model.generate(**inputs, max_new_tokens=128, do_sample=False, **kw)
    print(name, round(time.time() - t, 2), 's')
print(tok.decode(out[0], skip_special_tokens=True))

With greedy decoding both runs return the same text; the speculative one is usually faster on a single GPU.

More simulations

← Diffusion models · All AI Simulations · GRPO →

Last updated Nov 19, 2026
Type to search.
    ↑↓ navigate↵ open