KV cache
The KV cache stores attention keys and values from earlier tokens so a language model does not recompute them for every new token. Learn its role in speed and memory.
Overview
The KV cache is a memory store used when a transformer language model generates text. Such models write one token at a time. To choose each new token, every layer of the model looks back at all earlier tokens through attention. That look-back uses two vectors per earlier token in each layer: a key and a value. The KV cache keeps those keys and values after they are computed, so the model does not have to recompute them for every new token.
Without the cache, generating the thousandth token would mean reprocessing the previous 999 tokens from scratch. With it, the model computes keys and values only for the newest token and reuses everything else. This turns generation from a cost that grows with the square of the length into one that grows roughly linearly. The price is memory. The cache grows with every token, every layer and every request being served, and on long contexts it often uses more GPU memory than the model weights.
Picture a teacher marking a long exam. For each new answer, they need to remember what the student wrote earlier. Rereading the whole script before every answer would be slow. Instead they keep short notes in the margin as they go. The KV cache is those margin notes: quick to consult, but they fill the page as the exam gets longer.
What it is
| Part | What it holds or does |
|---|---|
| Query | Vector for the current token, used to look up earlier tokens |
| Key | Vector for each earlier token, compared with the query |
| Value | Vector for each earlier token, mixed into the output by attention weight |
| Cache tensor | Stored keys and values, per layer, per attention head, per token |
| Prefill phase | Processing the whole prompt at once and filling the cache |
| Decode phase | Generating tokens one by one, reading and extending the cache |
| Cache manager | Serving software that allocates, shares and frees cache memory |
Only keys and values are cached. The query is needed only for the current token, so it is not stored.
Where it came from
Caching keys and values follows directly from how decoder-only transformers work, so it has been part of efficient implementations since early models such as GPT-2. As models and context lengths grew, the cache became the main memory bottleneck in serving, and a series of ideas tackled it.
In 2019 Noam Shazeer proposed multi-query attention, where all attention heads share one set of keys and values, shrinking the cache sharply. In 2023 Google researchers introduced grouped-query attention, a middle ground in which groups of heads share keys and values; it was widely adopted in open models. Also in 2023, a team from UC Berkeley published PagedAttention, presented at SOSP 2023, and released it in the vLLM serving engine. PagedAttention stores the cache in fixed-size blocks, like pages in an operating system, which cut wasted memory and allowed many more requests to share a GPU.
Later work added prefix sharing between requests, as in the RadixAttention design of SGLang, compression of the cache into a smaller latent form, as in multi-head latent attention used by DeepSeek models, and low-precision cache storage. Hosted APIs now commonly offer prompt caching, which reuses the cache for repeated prompt prefixes and charges less for those tokens.
How it works
Step 1: Prefill the prompt
When a request arrives, the model processes all prompt tokens in parallel. In each layer it computes keys and values for every token and writes them into the cache. This phase is compute-heavy and fast per token, because GPUs handle large parallel work well.
Step 2: Decode one token
To produce the next token, the model computes the query, key and value for just that token. It appends the new key and value to the cache. It then compares the query with all cached keys, turns the scores into weights and mixes the cached values. This repeats in every layer.
Step 3: Repeat until done
Each further token adds one key and one value per layer and head. The decode phase is usually limited by memory bandwidth, not compute, because the GPU must read the whole cache for each new token.
Step 4: Estimate the memory
Cache size per token is: 2 (key and value) times number of layers times number of key-value heads times head dimension times bytes per number. For a model with 32 layers, 32 key-value heads, head dimension 128 and 16-bit numbers, that is 2 × 32 × 32 × 128 × 2 bytes, about 0.5 MB per token. A 4,000-token sequence needs about 2 GB for one request.
Step 5: Manage it in a server
Serving engines allocate cache blocks on demand, free them when requests finish, share identical prefixes between requests and, when memory runs short, pause or evict requests. Good cache management is a large part of why engines such as vLLM, SGLang and TensorRT-based servers achieve high throughput.
Step 6: Shrink it
Main ways to reduce cache size are fewer key-value heads (MQA, GQA), compressed latent caches, storing the cache in 8-bit or lower precision, sliding-window attention that keeps only recent tokens in some layers, and offloading older blocks to CPU memory.
Key concepts and terms
| Term | Meaning |
|---|---|
| Autoregressive decoding | Generating one token at a time, each based on the previous ones |
| Prefill | Processing the prompt and filling the cache |
| Decode | The token-by-token generation phase |
| Multi-query attention | All heads share one key and value set |
| Grouped-query attention | Groups of heads share key and value sets |
| PagedAttention | Storing the cache in fixed-size blocks to avoid waste |
| Prefix caching | Reusing cached keys and values for a shared prompt start |
| KV cache quantisation | Storing keys and values at lower precision |
| Eviction | Dropping cache entries to free memory |
| Time to first token | Latency until the first output token, dominated by prefill |
A worked example
An illustrative team serves an 8-billion-parameter model on one 80 GB GPU. The model uses grouped-query attention: 32 layers, 8 key-value heads, head dimension 128, cache stored in 16-bit precision.
- Per-token cache: 2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes, about 128 KB.
- Model weights in 16-bit: about 16 GB. Leaving room for activations and overhead, the team allows about 56 GB for the cache.
- Capacity: 56 GB divided by 128 KB is roughly 450,000 tokens of cache in total.
- Workload: each chat uses about 6,000 tokens of prompt and output. That allows about 75 concurrent conversations.
- With 8-bit cache storage, capacity roughly doubles to about 150 conversations, at a small accuracy cost they measure on their evaluation set.
- Every request starts with the same 2,000-token system prompt. With prefix caching, that part is stored once and shared, so it no longer counts against each conversation, and time to first token falls for repeat requests.
If the same model used full multi-head attention with 32 key-value heads, the per-token cache would be four times larger and capacity would fall to under 20 conversations. This is why attention design matters so much for serving.
Strengths and limitations
Strengths:
- Avoids recomputing past tokens, making generation practical.
- Enables prompt caching that cuts cost and latency for repeated prefixes.
- Well supported by all major serving engines.
- Works with any decoder-only transformer.
Limitations:
- Memory grows linearly with context length and with concurrent users.
- Decode speed is limited by reading the cache from memory.
- Long contexts can exhaust GPU memory and force smaller batches.
- Compression and eviction can reduce accuracy if pushed too far.
- Cache reuse depends on exact prefix matches; small prompt changes break it.
How it compares with alternatives
| Approach | Memory for past tokens | Speed per new token | Notes |
|---|---|---|---|
| Full KV cache | Grows with length | Fast | Standard for transformers |
| No cache, recompute | None stored | Very slow on long outputs | Only for tiny tests |
| MQA or GQA cache | Much smaller | Fast | Needs model trained this way |
| Sliding-window cache | Fixed per layer | Fast | Loses distant detail in those layers |
| State space models, for example Mamba | Fixed-size state | Fast | Different architecture, no growing cache |
Practical guidance and best practices
- Estimate cache memory with the formula before choosing hardware or batch size.
- Prefer models with grouped-query or latent attention for long-context serving.
- Use a serving engine with paged cache management, such as vLLM or SGLang.
- Put stable content at the start of prompts so prefix caching can reuse it; see context engineering.
- Test 8-bit cache storage on your own evaluation set before enabling it in production.
- Cap maximum output length to stop runaway generations from eating memory.
- Monitor cache utilisation, eviction rate and time to first token.
- Combine with speculative decoding and FlashAttention for further speed.
Common mistakes
- Budgeting GPU memory for model weights only and forgetting the cache.
- Putting timestamps or user IDs at the very start of the prompt, which breaks prefix reuse.
- Assuming a long advertised context window can be served at full concurrency on the same GPU.
- Quantising the cache aggressively without measuring quality.
- Confusing the KV cache with application-level response caching, which stores whole answers.
Frequently asked questions
Why are only keys and values cached, not queries?
Each token's query is used only when that token is being processed. Keys and values of earlier tokens are needed again for every later token, so they are worth storing.
Does the KV cache change the model's output?
No. A correct cache gives the same results as recomputing everything. Only lossy tricks, such as aggressive quantisation or eviction, can change outputs.
What is prompt caching in hosted APIs?
It is the provider reusing the cache for a prompt prefix that matches an earlier request. You usually pay less for cached input tokens and get a faster first token.
Why is decoding slower per token than prefill?
Prefill handles many tokens at once and keeps the GPU busy. Decoding handles one token per step and must read the whole cache from memory each time, so memory bandwidth becomes the limit.
Can the cache live on CPU or disk?
Yes, some systems offload older or idle cache blocks to CPU memory or storage and bring them back when needed. This saves GPU memory but adds transfer time.
How does batching interact with the cache?
Batching many requests improves GPU use, but each request needs its own cache. The cache size therefore often sets the maximum batch.
Glossary
- Attention: the mechanism that lets each token draw on earlier tokens.
- Attention head: one of several parallel attention computations in a layer.
- Head dimension: the length of each head's key and value vectors.
- Memory bandwidth: how fast data can be read from GPU memory.
- Block: a fixed-size chunk of cache memory in paged designs.
- Throughput: tokens or requests served per second.
- Context length: number of tokens in prompt plus output.
- Offloading: moving data from GPU to slower memory.
Summary
The KV cache stores the attention keys and values of earlier tokens so a transformer only has to compute the newest token at each step. It makes generation fast but grows with every token and every request, so it often dominates GPU memory in serving. Multi-query and grouped-query attention, PagedAttention in vLLM, prefix sharing, latent compression and low-precision storage are the main ways to tame it. Estimating cache size, choosing efficient attention designs and ordering prompts for prefix reuse are core skills for anyone serving large language models.
Related
It belongs to LLM concepts in my stack, under Generative AI and LLMs. Others in the same group: