What is Prompt caching? Definition and meaning
Prompt caching reuses the already-computed internal state for an identical prompt prefix across requests, so only the new part of each prompt has to be processed.
Definition
Prompt caching: Prompt caching reuses the already-computed internal state for an identical prompt prefix across requests, so only the new part of each prompt has to be processed.
In short: Never pay twice for a prefix.
When it is used
Chatbots with long system prompts, agents with large tool definitions, document question answering, coding assistants and any workload that repeats a long prompt start.
Where it fits
Prompt caching belongs to Frontier topics in the Generative AI and LLMs topic. These are the other terms in the same group:
- Test-time compute: Test-time compute is the extra computation a model spends while answering a single request, used to raise the quality of that answer.
- Diffusion language models: A diffusion language model generates text by iteratively denoising a whole sequence, usually by filling in masked tokens over several steps, instead of predicting one token at a time from left to right.
- LLM routing: LLM routing is the practice of choosing, for each incoming request, which language model or configuration should handle it.
- Model merging: Model merging creates a new model by combining the weights of two or more existing models, usually fine-tuned from the same base, without further gradient training.
- AI watermarking and provenance: Techniques that embed a hidden, detectable signal in AI-generated content, or attach signed records of its origin, so it can later be identified.
Learn more
The full guide to Prompt caching explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.
Previous term: Prometheus · Next term: Prompt engineering