Context engineering
Context engineering is the practice of deciding exactly what information, instructions, tools and memory go into a model's context at each step of an AI system.
Overview
Context engineering is the work of deciding what a language model sees each time it is called. A model has no memory between calls. Everything it knows about the task, the user, the available tools and the conversation so far must be placed into its context window as tokens. Context engineering is the discipline of choosing those tokens: which instructions, which documents, which tool definitions, which past messages and which notes from earlier steps.
It grew out of prompt engineering, but the scope is wider. A prompt is usually written once by a person. A context is assembled by software, often many times per task, from many sources. In an agent that runs for fifty steps, the system must decide at every step what to keep, what to drop, what to fetch and what to summarise. Good choices produce reliable behaviour. Poor choices produce confusion, cost and errors, even with a strong model.
Think of a surgeon's instrument tray. A good theatre nurse does not tip every tool in the hospital onto the tray. They lay out the right instruments, in a sensible order, and swap them as the operation moves on. Context engineering is that nurse's job for a language model.
What it is
A model's context is built from several kinds of content. Each competes for the same limited space and the same limited attention.
| Component | What it contains | Typical source |
|---|---|---|
| System instructions | Role, rules, tone and output format | Written by developers |
| Task input | The user's current request | User or upstream system |
| Retrieved knowledge | Documents, records or search results | Retrieval systems, databases |
| Tool definitions | Names, descriptions and schemas of callable tools | Tool registry, MCP servers |
| Tool results | Outputs from earlier tool calls | Runtime |
| Conversation history | Earlier turns of the dialogue | Session store |
| Memory | Durable facts, preferences and notes across sessions | Agent memory store |
| Scratchpad and plans | Intermediate notes the agent wrote for itself | The agent |
Context engineering covers four broad moves: selecting what goes in, ordering it, compressing it, and isolating parts of the work into separate contexts when one window is not enough.
Where it came from
The practice existed before the name. Builders of retrieval-augmented generation systems were already choosing chunks, and builders of early agents were already trimming history. Research made the problem visible. A 2023 study titled "Lost in the Middle" found that models used information at the start and end of long inputs better than information in the middle. Later work, including a study from Chroma Research, described "context rot": accuracy falling as inputs grow, well before the advertised window limit.
The term itself spread in mid-2025. In June 2025 Shopify's chief executive Tobi Lütke described context engineering as a core skill, LangChain's Harrison Chase wrote a post on its rise, and Andrej Karpathy described it as the art and science of filling the context window with just the right information for the next step. In September 2025 Anthropic published an engineering article on effective context engineering for AI agents. Since then the phrase has become the common label for this layer of AI system design.
How it works
Step 1: Define the job of each call
Start by stating what one model call must achieve. "Classify this ticket" needs far less context than "plan and carry out a data migration". A clear job tells you what is relevant.
Step 2: Write stable instructions
Put rules that never change into the system instructions. Keep them clear, specific and short. Explain the reason for important rules so the model can apply them sensibly to new cases.
Step 3: Retrieve just-in-time
Rather than loading every document at the start, give the model ways to fetch what it needs, such as search tools, file readers or database queries. Retrieval quality matters: see naive RAG, hybrid search and contextual retrieval.
Step 4: Curate tools
Every tool definition costs tokens and adds a choice the model must weigh. Offer a small, well-described set for each step. Overlapping tools with vague names cause wrong calls. See function calling.
Step 5: Manage history
Long sessions accumulate old tool outputs and stale turns. Common techniques are dropping raw tool results once they have been used, keeping only the last few turns verbatim, and summarising older parts. This is often called compaction.
Step 6: Persist notes outside the window
For long tasks, let the agent write notes, plans or progress files to external storage and read them back later. This keeps the window lean while preserving what matters. See long-running tasks.
Step 7: Isolate with sub-agents
When a task has independent parts, run each in its own fresh context through sub-agents. Each returns a short result, so the main agent never carries the full detail of every branch.
Step 8: Order for attention and caching
Place stable content first and changing content last. This helps prefix caching, which reuses the KV cache for identical prompt starts, and puts the current question near the end where models attend well.
Key concepts and terms
| Term | Meaning |
|---|---|
| Context window | The maximum number of tokens a model can read in one call |
| Context budget | The share of the window you allow each component to use |
| Compaction | Summarising or pruning history to free space |
| Just-in-time retrieval | Fetching data when needed instead of preloading it |
| Context rot | Degrading accuracy as the input grows longer |
| Lost in the middle | The tendency to under-use information placed mid-context |
| Tool sprawl | Too many overlapping tools in one context |
| Context isolation | Splitting work across separate model contexts |
| Prefix caching | Reusing computation for an identical prompt start |
| Progressive disclosure | Showing short summaries first and loading detail on demand, as in skills |
A worked example
Consider an illustrative customer-support agent with a 200,000-token window. Its first version loaded everything at once:
- System prompt and policies: 12,000 tokens.
- All 40 tool definitions: 18,000 tokens.
- The customer's full order history: 30,000 tokens.
- Twenty help articles from a keyword search: 45,000 tokens.
- The full conversation so far: 25,000 tokens.
That is 130,000 tokens per call. Answers were slow, costly and sometimes cited the wrong article.
The redesign applied context engineering:
- Policies were trimmed and restructured to 5,000 tokens.
- A router step picked one of four tool groups, so each call saw about 8 tools, roughly 3,500 tokens.
- Order history was replaced by a lookup tool. The agent fetched only the order in question, about 800 tokens.
- Retrieval returned the top 4 reranked passages instead of 20 full articles, about 3,000 tokens.
- Older conversation turns were summarised into 600 tokens, with the last 4 turns kept verbatim, about 2,000 tokens.
The new context was about 15,000 tokens, roughly nine times smaller. In this example, the team's evaluation set showed fewer wrong-article citations and lower latency. The model did not change. Only what it saw changed.
Strengths and limitations
Strengths:
- Improves accuracy without changing or retraining the model.
- Cuts cost and latency by removing unneeded tokens.
- Makes long-running agents possible beyond a single window.
- Gives clear, testable levers: what to retrieve, which tools, how much history.
- Works with any model provider.
Limitations:
- Adds engineering complexity: routers, summarisers, stores and caches.
- Summaries can lose details that later turn out to matter.
- Retrieval errors silently starve the model of needed facts.
- Hard to evaluate without good traces and test sets.
- Best settings change when the model or the task changes.
How it compares with alternatives
| Approach | What it changes | When to prefer it |
|---|---|---|
| Context engineering | What the model sees at each call | Agents, RAG, multi-step systems |
| Prompt engineering | Wording of instructions and examples | Single, well-defined calls |
| Fine-tuning, for example LoRA | The model's weights | Stable, repeated tasks with a lot of data |
| Bigger context window alone | How much fits | Quick prototypes; rarely a full fix |
| Agent memory | What persists across sessions | Personalised or long-lived assistants |
These approaches stack. Most production systems combine careful prompts, engineered context and, sometimes, a tuned model.
Practical guidance and best practices
- Treat the context window as a budget. Assign rough token limits to each component.
- Log the exact context sent on every call, so you can see what the model actually saw. See tracing.
- Prefer tools that fetch data over pasting data in advance.
- Keep tool sets small per step and write descriptions as carefully as instructions.
- Clear raw tool outputs once their result has been used or summarised.
- Put stable content first to benefit from prefix caching.
- Use sub-agents for wide searches and pass back only condensed findings.
- Build an evaluation set and measure each change; see evals.
- Revisit the design after every model upgrade, since newer models may need less scaffolding.
Common mistakes
- Filling the window "just in case", which dilutes attention and raises cost.
- Exposing dozens of similar tools and then blaming the model for wrong choices.
- Summarising too aggressively and losing identifiers, numbers or user constraints.
- Placing the actual question in the middle of a long block of documents.
- Mixing instructions inside retrieved text, which also invites prompt injection.
- Never inspecting real contexts, so problems stay invisible.
Frequently asked questions
Is context engineering just a new name for prompt engineering?
It overlaps, but the focus differs. Prompt engineering is mostly about how you phrase instructions. Context engineering is about the whole assembly of information, tools and memory, usually built by software and changing at each step.
If models now have very large windows, why bother?
Large windows help, but accuracy and speed still fall as inputs grow, and every token costs money. A smaller, relevant context usually beats a huge, noisy one.
What is compaction?
Compaction means shrinking the conversation or work history, usually by summarising older parts and dropping raw tool outputs, so the agent can keep working without running out of space.
How do I know what to include?
Start from the job of the call and ask what the model needs to do that job well. Then test. Remove a component and see if quality drops. If it does not, leave it out.
Does context engineering help with security?
Yes, partly. Keeping untrusted content clearly separated from instructions, and limiting which tools are available, reduces the impact of injected text. It does not replace other defences.
Who should own context engineering in a team?
Usually the engineers who build the AI system, working with domain experts who know what information matters. It sits between application code, data retrieval and model behaviour.
Glossary
- Token: a unit of text, roughly a word piece, counted against the window.
- System prompt: standing instructions given to the model on every call.
- Retrieval: fetching relevant documents or records for a query.
- Router: a step that decides which tools or path to use.
- Sub-agent: a separate model call with its own fresh context for a sub-task.
- Scratchpad: notes the agent writes for its own later use.
- Prefix cache: stored computation for a repeated prompt start.
- Reranking: re-ordering retrieved results by relevance.
Summary
Context engineering is the practice of deciding exactly what goes into a model's context window at each step: instructions, retrieved knowledge, tools, history and memory. The name spread in mid-2025, but it builds on earlier lessons from retrieval systems and research showing that long, noisy inputs hurt accuracy. The core techniques are to budget tokens, retrieve just in time, keep tool sets small, compact history, store notes outside the window and isolate work in sub-agents. Done well, it makes the same model faster, cheaper and more reliable.
Related
It belongs to LLM concepts in my stack, under Generative AI and LLMs. Others in the same group: