The KV cache: why generation needs it and what it costs
Every generated token attends to all the tokens before it. Recomputing their keys and values every step would be quadratic; caching them makes decoding linear, at a memory cost that can exceed the model itself.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The upper chart is cache memory against context length for three attention types, with the weights for comparison. Below, the prompt is processed in one pass and then one token is generated per step; each step adds one row of keys and values. The side chart counts the key/value projections with and without the cache.
Things to try
- Pick the 70B-class model, 128k context and batch 32.
- Switch from GQA to MHA and back.
- Generate tokens and compare the two work curves.
How it works in real systems
vLLM's PagedAttention stores the cache in fixed-size blocks; serving stacks quantise it to fp8, share it across requests with the same prefix (prompt caching) and evict it under memory pressure.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Size the cache before you deploy
def kv_cache_bytes(layers, kv_heads, head_dim, tokens, batch=1, bytes_per_value=2):
return 2 * layers * kv_heads * head_dim * tokens * batch * bytes_per_value # 2 = keys and values
gb = kv_cache_bytes(layers=32, kv_heads=8, head_dim=128, tokens=8192, batch=8) / 2**30
print(f"{gb:.1f} GB") # 8B-class model with grouped-query attention: 8.0 GB for 8 × 8k tokens
Step 2 · Decode with and without the cache
past_key_values carries the cached keys and values; with it each step feeds only the newest token.
import torch, time
from transformers import AutoTokenizer, AutoModelForCausalLM
name = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(name); model = AutoModelForCausalLM.from_pretrained(name).eval()
ids = tok("Explain the KV cache in one sentence:", return_tensors="pt").input_ids
@torch.no_grad()
def greedy(ids, n, cache=True):
out = model(ids, use_cache=cache); past = out.past_key_values if cache else None
for _ in range(n):
nxt = out.logits[:, -1:].argmax(-1)
ids = torch.cat([ids, nxt], 1)
out = model(nxt, past_key_values=past, use_cache=True) if cache else model(ids, use_cache=False)
if cache: past = out.past_key_values
return ids
for c in (True, False):
t = time.time(); r = greedy(ids, 64, c); print("cache" if c else "no cache", round(time.time() - t, 2), "s")
print(tok.decode(r[0]))
Step 3 · Serve it efficiently
Production servers manage the cache for you: vLLM stores it in pages (PagedAttention) and batches requests continuously; the cache can also be quantised.
pip install vllm
vllm serve Qwen/Qwen2.5-0.5B-Instruct --max-model-len 8192 --gpu-memory-utilization 0.9 --kv-cache-dtype fp8