# Prompt caching

Prompt caching stores the computed attention state for the start of a prompt so later requests with the same prefix skip that work, cutting cost and time to first token.

## Overview

Many applications send a language model nearly the same long prompt again and again. A support chatbot repeats a 5,000-token system prompt with every message. An agent resends dozens of tool definitions on every turn. A document assistant asks many questions about the same 50-page contract. Without help, the model recomputes its internal state for all of that repeated text every single time, and you pay for it every single time.

Prompt caching removes that waste. When a request begins with exactly the same tokens as an earlier request, the serving system reuses the internal state it already computed for those tokens and only processes the new part at the end. The answer is the same as it would have been without the cache, but the request costs less and starts responding sooner. Hosted model providers sell this as a feature with discounted pricing for cached tokens. Self-hosted inference engines such as [vLLM](/tools/vllm) and [SGLang](/tools/sglang) do the same thing automatically under the name prefix caching.

An everyday analogy: imagine a translator who is handed the same 40-page contract every morning with a different question stapled to the back. The first time, they read the whole contract carefully. If they keep their notes, the next morning they can skip straight to the new question. If anyone changes a single word on page one, though, the notes can no longer be trusted, and they must start reading again. Prompt caching works the same way: reuse is only safe when the beginning is identical.

## What it is

| Part | What it does |
|---|---|
| Prefix | The leading part of a prompt that stays identical across requests |
| [KV cache](/tools/kv-cache) | The stored attention keys and values computed for each token |
| Cache key | A hash of the exact prefix tokens (plus any settings that affect them) |
| Breakpoint | A point in the prompt where a cache entry is written or looked up |
| Cache write | The first request that computes and stores a prefix |
| Cache hit (read) | A later request that reuses a stored prefix |
| Time to live (TTL) | How long an unused entry is kept before it expires |
| Eviction policy | Rules for discarding entries when memory runs short, often least recently used |

The thing being cached is not the text and not the answer. It is the key and value tensors that the [attention mechanism](/tools/attention-mechanism) computes for each token during prefill, the phase in which the model reads the prompt. These tensors are exactly what a model normally keeps during generation so that it does not reprocess earlier tokens for every new one. Prompt caching extends that idea across separate requests.

Two delivery styles exist. In implicit or automatic caching, the system detects shared prefixes by itself. In explicit caching, the developer marks where cacheable content ends, or creates a named cache object, and gets more predictable behaviour in return. Hosted providers offer one or both styles, with different rules.

## Where it came from

Prompt caching grew out of the KV cache that every [transformer](/tools/transformers) decoder uses during generation. The serving breakthrough that made reuse across requests practical was PagedAttention, introduced by Woosuk Kwon and colleagues at UC Berkeley in 2023 in "Efficient Memory Management for Large Language Model Serving with PagedAttention", the paper behind vLLM. By storing the KV cache in fixed-size blocks, like pages in an operating system, it became straightforward to share identical blocks between requests.

In December 2023, Lianmin Zheng and colleagues released SGLang, later published at NeurIPS 2024, which introduced RadixAttention. It stores cached prefixes in a radix tree so that many requests sharing different-length prefixes can reuse as much as possible; the paper reported up to 6.4 times higher throughput than existing systems across a range of workloads. vLLM added automatic prefix caching based on hashing blocks of tokens.

Hosted providers followed. Google introduced context caching for the Gemini API in May 2024. Anthropic introduced prompt caching for its API in August 2024 with explicit cache markers. OpenAI announced automatic prompt caching on 1 October 2024, applying to prompts longer than 1,024 tokens and pricing cached input at half the normal rate. On 8 May 2025, Google added implicit caching for Gemini 2.5 models, passing on a 75 per cent discount automatically on cache hits. Pricing, minimum lengths and lifetimes have changed several times since, so the current documentation is always the authority.

## How it works

### Prefill, decode and why prefixes matter

Serving a request has two phases. Prefill processes every prompt token and builds the KV cache. Decode generates the answer one token at a time, reading that cache. For long prompts and short answers, prefill dominates both cost and the time before the first output token appears. Prompt caching skips prefill for the part of the prompt already in the cache.

The reason only prefixes can be reused is causal attention. Each token's keys and values depend on every token before it. If token 10 changes, the stored state for tokens 11 onwards is no longer valid, even if those later tokens are identical. A cache is therefore matched from the start of the prompt and stops at the first difference.

### Hashing blocks

Self-hosted engines usually divide the token sequence into fixed-size blocks and give each full block a hash. In vLLM's design, the hash combines the parent block's hash, the block's token IDs, and extra keys such as a LoRA adapter ID, multimodal input hashes or a cache salt. Chaining the parent hash means a block's identity encodes the entire prefix before it. A new request is hashed block by block; every leading block whose hash is already stored is reused, and computation starts at the first miss. Unused blocks sit in a queue and are evicted least recently used first.

### The cost arithmetic

```text
Cost without caching:  C₀ = N_total × p
Cost with caching:     C  = N_write × p × w  +  N_read × p × r  +  N_new × p
Break-even reuse:      a prefix pays off once (w − 1) < k × (1 − r)
```

Here N_total is the number of input tokens in a request and p is the normal price per input token. N_write is the number of tokens written to the cache, N_read the number read from it, and N_new the uncached tokens after the cached prefix. The factor w is the provider's write multiplier and r its read multiplier, both relative to the normal input price. In the last line, k is the number of times the prefix is reused before it expires: caching pays off when the extra paid on the write is smaller than the savings across k reads.

### Provider rules at the time of writing

Each provider documents its own rules, and they change. As of November 2026:

- **Anthropic** uses a `cache_control` marker on a content block, or a single top-level setting for automatic placement, with up to four breakpoints per request. The prefix is built in the order tools, then system, then messages, so a change to tool definitions invalidates everything after it. The default lifetime is 5 minutes, refreshed on each use, with an optional 1-hour lifetime. Cache writes cost 1.25 times the base input price for 5 minutes or 2 times for 1 hour, and reads cost 0.1 times for most models, with lower read multipliers on some newer models. Minimum cacheable lengths vary by model, from 512 to 4,096 tokens.
- **OpenAI** caches by default on supported models. For its newest models the minimum is 1,024 visible input tokens, the default retention is at least 30 minutes with an optional 12-hour setting, writes are charged at 1.25 times or 2.5 times respectively, and reads at 0.1 times for most models. Cached state lives on individual machines, and routing depends on a hash of the prompt's beginning and an optional `prompt_cache_key`. Older models use different retention options.
- **Google Gemini** applies implicit caching automatically on Gemini 2.5 and newer models, with minimum sizes from 2,048 to 4,096 tokens depending on the model. Explicit caching, where you create a cache object with a lifetime, remains available for some models through the generateContent API.

### Self-hosted prefix caching

In vLLM, automatic prefix caching is controlled by the `enable_prefix_caching` engine option, and its documentation notes that it speeds up prefill only, not decoding. SGLang's RadixAttention keeps a tree of cached prefixes and is designed for workloads such as multi-turn chat, few-shot prompting and agent loops where many requests share partial prefixes. For multi-tenant systems, vLLM supports a per-request cache salt so that only requests in the same trust group can share cached blocks, which limits timing attacks that infer someone else's cached content from response speed.

### A code example

The example below uses the Anthropic Python SDK to mark a long, stable system prompt as cacheable and then reads the cache counters from the response.

```python
import anthropic

client = anthropic.Anthropic()
long_policy = open("support_policy.md").read()  # thousands of tokens, unchanged between calls

response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=512,
    system=[
        {"type": "text", "text": "You are a support assistant."},
        {"type": "text", "text": long_policy, "cache_control": {"type": "ephemeral"}},
    ],
    messages=[{"role": "user", "content": "Can I return an opened item?"}],
)

u = response.usage
print(u.cache_creation_input_tokens, u.cache_read_input_tokens, u.input_tokens)
```

On the first call, `cache_creation_input_tokens` shows the written prefix. On a second call within the lifetime, `cache_read_input_tokens` should show the same prefix being reused, and only the new question counts as ordinary input.

## Key concepts and terms

| Term | Meaning |
|---|---|
| Prefill | Processing all prompt tokens to build the KV cache |
| Decode | Generating output tokens one at a time |
| Time to first token (TTFT) | Delay before the first output token arrives |
| Prefix | The identical leading part of prompts that can be reused |
| Breakpoint | A marked position where a cache entry ends |
| Cache write | Storing a newly computed prefix, often at a premium price |
| Cache read | Reusing a stored prefix, usually at a discount |
| TTL | How long an unused cache entry lives |
| Implicit caching | Caching that happens automatically without markers |
| Explicit caching | Caching controlled by developer markers or cache objects |
| RadixAttention | SGLang's tree-based prefix reuse technique |
| Cache salt | A per-tenant value mixed into hashes to isolate caches |

## A worked example

Take an illustrative internal assistant that answers questions about a 20,000-token policy handbook. Each request is the handbook plus a short instruction block, followed by a 200-token question. Staff send about 300 questions an hour during the working day, spread fairly evenly.

Without caching, each request processes roughly 20,200 input tokens. With a cache read multiplier of 0.1 and a 5-minute write multiplier of 1.25, the arithmetic changes sharply. The first request writes the 20,000-token prefix and pays 1.25 times for it. Every request in the next few minutes reads the prefix at a tenth of the price and pays full price only for its 200-token question. With five requests a minute, the cache is touched far more often than every 5 minutes, so it stays warm all day and the write cost is paid rarely.

Roughly, the input cost per request falls from about 20,200 token-units to about 2,000 for the cached part plus 200 for the question, a little over a tenth of the original. The time to first token also drops, because the model no longer has to process 20,000 tokens before starting its answer.

Then someone adds the current date and time to the top of the system prompt. The cache hit rate falls to zero overnight, because the very first tokens now differ on every request. The team moves the timestamp to the end of the prompt, after the cached handbook, and the savings return. This is the most common way prompt caching silently stops working.

## Strengths and limitations

**Strengths:**

- Large cost reductions on repeated long prefixes, at the discount rates each provider publishes.
- Lower time to first token, because the cached part of prefill is skipped.
- No change to output: the response is the same as without caching.
- Works naturally with chat history, agent loops, few-shot examples and document question answering.
- In self-hosted engines, frees GPU capacity for more requests, raising throughput.

**Limitations:**

- Only exact prefixes are reused; one changed token near the start invalidates everything after it.
- Hosted caches expire after a short lifetime and are best-effort, so hits are not guaranteed.
- Writes can cost more than ordinary input, so rarely reused prefixes may cost more with caching.
- Short prompts below the provider's minimum are not cached at all.
- Decoding speed is unchanged; long outputs see little benefit.
- Shared caches in multi-tenant systems can leak information through timing unless isolated.

## How it compares with alternatives

| Technique | What is reused | Same output guaranteed? | Typical use |
|---|---|---|---|
| Prompt caching | Computed state for an identical prefix | Yes | Long repeated prompts |
| Semantic or response caching | A previous answer to a similar question | No | Frequently repeated questions |
| [KV cache](/tools/kv-cache) within a request | State for earlier tokens during generation | Yes | Every autoregressive generation |
| [Context engineering](/tools/context-engineering) | Nothing; shrinks what is sent | Changes the prompt | Reducing tokens overall |
| [Fine-tuning](/tools/fine-tuning) | Knowledge moved into weights | Changes the model | Removing long instructions entirely |

Response caching returns a stored answer, which is cheaper still but risks giving a stale or mismatched reply. Prompt caching is safer because the model still generates a fresh answer. Fine-tuning can remove the need for a long prompt altogether, but it is a far bigger commitment and harder to update than editing a cached system prompt.

## Practical guidance and best practices

1. Order prompts from most stable to least stable: tools and system instructions first, reference documents next, conversation history after that, and the newest user message last.
2. Keep the cached prefix byte-for-byte identical. Avoid timestamps, request IDs, random examples or reordered tools near the start.
3. Serialise JSON with a fixed key order; some languages randomise map ordering, which silently breaks caches.
4. Place breakpoints on the last block that stays identical, not on content that changes every call.
5. Check the cache counters in every response and alert when the hit rate drops.
6. Choose the lifetime by traffic pattern: short default lifetimes suit steady traffic, longer paid lifetimes suit bursty or infrequent use.
7. For parallel requests sharing a new prefix, send one first and wait for it to start responding so that the others can hit the cache.
8. Do the break-even sum before paying for longer lifetimes or caching rarely reused content.
9. In self-hosted, multi-tenant deployments, isolate caches per tenant with salts or separate pools, and link cache metrics into your [cost and latency budgets](/tools/cost-and-latency-budgets).

## Common mistakes

- Putting dynamic content, such as the date or the user's name, at the top of the system prompt.
- Changing tool definitions between turns of an agent, which invalidates the whole cache.
- Expecting short prompts to be cached when they fall below the provider's minimum length.
- Assuming a cache hit is guaranteed; hosted caches are best-effort and may be evicted early.
- Measuring savings on a single test request instead of real traffic with realistic gaps between calls.
- Treating prompt caching as a response cache and expecting identical answers to be returned without generation.

## Frequently asked questions

### Does prompt caching change the model's answer?

No. The cached keys and values are the same numbers the model would have computed anyway, so the response is the same as without caching. Any variation you see comes from normal sampling randomness, not from the cache.

### Why is my cache hit rate zero?

Usually because something near the start of the prompt changes on every request, such as a timestamp, a random example or a reordered tool list. Other causes are prompts below the minimum length, gaps between requests longer than the cache lifetime, or settings changes that the provider treats as part of the cache key.

### Is prompt caching the same as semantic caching?

No. Semantic caching stores previous answers and returns one when a new question looks similar enough. Prompt caching stores the model's internal state for an identical prefix and still generates a new answer. Semantic caching saves more but can return wrong or stale answers.

### Does caching help with long outputs?

Only a little. Caching skips prefill for the cached prefix, which reduces cost and time to first token. It does not speed up the generation of output tokens, so for short prompts with long answers the benefit is small.

### Is it safe to share caches between users?

Hosted providers scope caches so that customers do not share them; OpenAI, for example, documents that caches are not shared across organisations. Check your provider's documentation for the exact scope. In your own multi-tenant deployment, a shared cache can leak information through timing differences, so isolate tenants with cache salts or separate pools.

### Should I always use the longest cache lifetime?

No. Longer lifetimes usually cost more to write. If your traffic touches the prefix every few minutes, the default lifetime keeps it warm for free. Longer lifetimes pay off when requests are spread out, for example a document reread every half hour.

## Glossary

- Prompt caching: reusing computed state for an identical prompt prefix across requests.
- Prefix caching: the self-hosted engine term for the same idea.
- KV cache: stored attention keys and values for processed tokens.
- Prefill: the phase in which the prompt is processed.
- Breakpoint: the marked end of a cacheable prefix.
- TTL: the lifetime of an unused cache entry.
- Cache hit rate: the share of input tokens served from cache.
- Cache salt: a value mixed into cache keys to keep tenants separate.
- RadixAttention: tree-based prefix sharing in SGLang.
- Time to first token: the wait before the first output token appears.

## Summary

Prompt caching saves the computed attention state for the beginning of a prompt and reuses it when later requests start with exactly the same tokens. It rests on the KV cache and on paged, block-based memory management in serving engines, and it is offered by hosted providers as discounted cached input and by engines such as vLLM and SGLang as prefix caching. It reduces cost and time to first token without changing outputs, but only for identical prefixes, only within a lifetime, and only above a minimum length. The practical rules are simple: put stable content first and changing content last, keep the prefix byte-for-byte identical, watch the cache counters, and check each provider's current documentation, because prices and limits change.