# The KV cache: why generation needs it and what it costs

Every generated token attends to all the tokens before it. Recomputing their keys and values every step would be quadratic; caching them makes decoding linear, at a memory cost that can exceed the model itself.

<div data-sim="kvcache"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

The upper chart is cache memory against context length for three attention types, with the weights for comparison. Below, the prompt is processed in one pass and then one token is generated per step; each step adds one row of keys and values. The side chart counts the key/value projections with and without the cache.

## Things to try

- Pick the 70B-class model, 128k context and batch 32.
- Switch from GQA to MHA and back.
- Generate tokens and compare the two work curves.

## How it works in real systems

vLLM's PagedAttention stores the cache in fixed-size blocks; serving stacks quantise it to fp8, share it across requests with the same prefix (prompt caching) and evict it under memory pressure.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · Size the cache before you deploy

```python
def kv_cache_bytes(layers, kv_heads, head_dim, tokens, batch=1, bytes_per_value=2):
    return 2 * layers * kv_heads * head_dim * tokens * batch * bytes_per_value   # 2 = keys and values
gb = kv_cache_bytes(layers=32, kv_heads=8, head_dim=128, tokens=8192, batch=8) / 2**30
print(f"{gb:.1f} GB")      # 8B-class model with grouped-query attention: 8.0 GB for 8 × 8k tokens
```

### Step 2 · Decode with and without the cache

`past_key_values` carries the cached keys and values; with it each step feeds only the newest token.

```python
import torch, time
from transformers import AutoTokenizer, AutoModelForCausalLM
name = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(name); model = AutoModelForCausalLM.from_pretrained(name).eval()
ids = tok("Explain the KV cache in one sentence:", return_tensors="pt").input_ids

@torch.no_grad()
def greedy(ids, n, cache=True):
    out = model(ids, use_cache=cache); past = out.past_key_values if cache else None
    for _ in range(n):
        nxt = out.logits[:, -1:].argmax(-1)
        ids = torch.cat([ids, nxt], 1)
        out = model(nxt, past_key_values=past, use_cache=True) if cache else model(ids, use_cache=False)
        if cache: past = out.past_key_values
    return ids
for c in (True, False):
    t = time.time(); r = greedy(ids, 64, c); print("cache" if c else "no cache", round(time.time() - t, 2), "s")
print(tok.decode(r[0]))
```

### Step 3 · Serve it efficiently

Production servers manage the cache for you: vLLM stores it in pages (PagedAttention) and batches requests continuously; the cache can also be quantised.

```bash
pip install vllm
vllm serve Qwen/Qwen2.5-0.5B-Instruct --max-model-len 8192 --gpu-memory-utilization 0.9 --kv-cache-dtype fp8
```

<div class="cl-end"></div>

## More simulations

[← Positional encoding](/ai-simulations/positional-encoding) · [All AI Simulations](/ai-simulations) · [Beam search →](/ai-simulations/beam-search)
