Context has a price: why trimming a coding agent's prompt can double its bill
A new arXiv study of more than 20,000 coding-agent runs finds that adding retrieved code or deleting old tool output often makes runs more expensive once cached input is cheap. Here is how prefix caching shapes the bill and how to measure it.
- Reading time
- 17 min
- Sections
- 8
- Code examples
- 1
- Tables
- 2
- Sources
- 9
- Published
- 19 Nov 2026
- What changed
- "Context Has a Price" tests context tricks on coding agents
- Why it matters
- On hosted APIs, cache reads are cheap, so the cheapest history is often the one you never edit.
- What to do
- Log cached, uncached and output tokens for every call
Most advice about coding agents says the same thing: keep the context lean. Retrieve the right code up front so the agent searches less. Delete old tool output so each prompt is shorter. Both ideas sound like they should save money.
A paper submitted to arXiv on 17 November 2026 tests those ideas at scale and finds that the answer depends on one detail of the bill: how much cheaper cached input is than fresh input. When cached input costs a tenth of the normal price, adding retrieved code made runs more expensive, and removing old observations sometimes more than doubled the cost. The agents also reacted to missing context by taking more steps.
Many teams judge a context trick by how many tokens it removes from one prompt. The study shows that the right unit is the whole task, priced with all three parts of the token bill, and measured without other experiments warming the same cache.
What happened
"Context Has a Price" tests context tricks on coding agents (17 November 2026)
Zerui Yang and eleven co-authors posted "Context Has a Price: Adding and Removing Context for LLM Coding Agents under Prefix Caching" (arXiv 2611.22238). The authors report more than 20,000 runs of open models on SWE-bench Verified, 80 SWE-bench test tasks outside Verified, and 224 Loc-Bench tasks. The main model is Qwen3.6-27B, alongside Qwen3-Coder-30B-A3B, smaller Qwen3.5 models, Gemma-4-26B-A4B and GLM-4.7-Flash, all served with vLLM with prefix caching on. The scaffolds were a minimal bash agent, mini-SWE-agent 2.4.6 and SWE-agent 1.1.0.
Each run is priced as uncached input, plus a cache price times cached input, plus an output price times output. The main settings are a cache price of 0.1 (a tenth of uncached input) and 1 (no discount), with output at four times the input price. The authors report that:
- Retrieved code cut read and search commands by 18 to 34 percent, yet at a one-tenth cache price runs cost 33 to 42 percent more, with no measurable gain in resolution.
- Masking old tool output saved 19 to 36 percent with no cache discount, but raised cost by 38 to 116 percent at a one-tenth cache price.
- Shorter prompts do not guarantee cheaper runs, because agents may take more steps.
- Concurrent experiments that share cached prefixes can distort cost comparisons.
A same-week paper stores the Gemma 4 global KV cache once (17 November 2026)
On the same day, Kushagra Bainsla posted "Storing the Gemma 4 K=V Cache Once" (arXiv 2611.23035). In Gemma 4 31B's global attention layers the key and value are tied, and the paper notes that the Gemma 4 report says this can shrink the global KV cache by up to 37.5 percent. The author found that the serving engines checked still store a full key and value, 1,024 numbers per KV head per token. The vLLM prototype stores one 640-number row and reads it through two overlapping views with an unchanged attention kernel. The author reports decode KV memory per request falling by a factor of 1.34 to 1.40 at 32K to 64K tokens, with eager decode steps 4 to 9 percent slower and no 8-bit KV cache support yet. It is a reminder that the cache also costs server memory, which limits how many prefixes stay cached.
How prefix caching works
A transformer reads a prompt token by token. For each token, every attention layer computes a key and a value vector. Later tokens look back at those vectors. Together they form the KV cache. Computing it for a long prompt is the "prefill" step, and it is most of the work for long inputs. You can watch the mechanism step by step in the KV cache simulation.
Here is the key point. The key and value for token number 500 depend on tokens 1 to 499, because attention mixes in everything before it. So the cache for a prompt is only valid for that exact sequence of earlier tokens. If two requests start with the same tokens, the second can reuse the first one's cache for that shared start and compute only the new tail. That shared start is the prefix.
vLLM's design document shows how a server tracks this. It splits the cache into fixed-size blocks and gives each full block a hash built from three parts: the hash of the block before it, the exact token IDs in the block, and extra inputs such as a LoRA ID or a tenant salt. Because each hash includes its parent's hash, a block's identity encodes the whole prefix that led to it. Change one token in block 3 and blocks 3, 4, 5 and onward all get new hashes. Old blocks that no request still uses are evicted in least-recently-used order. vLLM also notes that prefix caching speeds up only prefill, not output generation.
An everyday analogy: a train of coupled carriages. Adding carriages at the back is cheap. Swapping carriage 3 means uncoupling everything behind it and rebuilding from there.
Hosted APIs expose the same rule with billing attached. Anthropic's documentation says the cached prefix covers tools, then system prompt, then messages, and a change at one level invalidates it and every level after. OpenAI's guide says the entire rendered prefix must match and warns that summarizing or truncating earlier turns can reset reuse. Google says Gemini implicit caching is on by default for Gemini 2.5 and newer and advises putting reusable content first. The site's serving stack article covers how servers route requests to the replica that already holds a prefix; this article stays with what the prefix means for an agent's bill.
The three-part token bill
Every agent step sends a prompt and gets a reply. The bill for that step has three parts.
| Part | What it is | Typical relative price | What makes it grow |
|---|---|---|---|
| Uncached input | Prompt tokens the server must prefill from scratch | 1 (the base rate) | New tool output, any edit early in the prompt, cache expiry |
| Cached input | Prompt tokens served from a stored prefix | About 0.1 on many current APIs; lower on some models | Long, stable history that grows by appending |
| Output | Tokens the model generates | Several times the input rate; the paper uses 4 | More steps, longer reasoning, longer edits |
| Cache write (some APIs) | Premium for storing a new prefix | 1.25 times base for short lifetimes on Anthropic and OpenAI | Each time the prefix changes and must be stored again |
Anthropic lists cache reads at 0.1 times base input for most models, 0.05 for Claude Opus 5.5 and Sonnet 5.5, and 0.025 for some others. OpenAI lists 0.1 on GPT-5.6 and later and 0.05 on certain GPT-6.x models. The paper's formula leaves out write premiums, so on APIs that charge them, breaking the cache likely costs somewhat more than its numbers suggest. That is my inference, not a study result.
A worked example (illustrative)
The numbers below are made up to show the arithmetic. They are not from the paper. Assume a price of $1 per million uncached input tokens, cached input at 0.1 of that, and output at 4 times.
Picture an agent whose first prompt is 10,000 tokens: system prompt, tool definitions and the issue. Each step adds 2,000 tokens of new tool output and agent messages, and the model writes 200 output tokens per step.
At step 10, the prompt is 10,000 + 9 × 2,000 = 28,000 tokens. Everything except the newest 2,000 tokens matches step 9's prompt, so 26,000 tokens are cached. The step costs:
- Uncached: 2,000 × $1 per million = $0.0020
- Cached: 26,000 × $0.10 per million = $0.0026
- Output: 200 × $4 per million = $0.0008
- Total: $0.0054
Without a cache discount, the same step would cost 28,000 × $1 + 200 × $4 per million = $0.0288, more than five times as much. With caching, a long history that only grows at the end is cheap.
Now run the whole task under three designs, with the last one at two step counts. I computed these with a short script using the assumptions above.
| Design (illustrative) | Uncached tokens | Cached tokens | Cost at 0.1 cache price | Cost with no discount |
|---|---|---|---|---|
| Append only, 30 steps | 68,000 | 1,102,000 | $0.202 | $1.194 |
| Add 14,000 tokens of retrieved code to the prefix, 27 steps | 76,000 | 1,274,000 | $0.225 (+11%) | $1.372 (+15%) |
| Rolling masking: keep the last 10 observations in full, replace older ones with 50-token stubs, 30 steps | 410,950 | 388,550 | $0.474 (+134%) | $0.824 (−31%) |
| Rolling masking, but the agent needs 33 steps | 471,100 | 421,550 | $0.540 (+167%) | $0.919 (−23%) |
The toy shows the paper's pattern. Masking removes tokens, so it wins when every token costs the same. But each time the window slides, a stub replaces an early observation and every token after it becomes uncached again. At a one-tenth cache price that rebuild costs far more than the tokens saved, and extra steps widen the gap.
Why compaction and retrieval interact with cache hits
Removing context: masking and summarization
Observation masking replaces old tool output with a short placeholder. Summarization rewrites part of the history into a shorter text. Both edit the middle of the prompt, which is exactly what breaks a prefix cache.
The authors report that rolling masking of all but the latest ten observations saved 19 to 25 percent without a cache discount, but raised cost by 116 and 87 percent for two Qwen models at a one-tenth price. They also derive a "break-even" rule that masks only when expected savings exceed the cache-rebuild cost. On the main setup it cost 2.8 and 1.8 percent more at a one-tenth price and 26 and 21 percent less without a discount: a large loss became a small one, not a saving.
The second effect is behaviour. Masked SWE-agent runs grew from 92 to 116 steps, and masking added 1.5 to 3.0 reads and 1 to 2 searches per Qwen3.6-27B run. With actions held fixed, the authors estimate masking would have saved that model 3 to 22 percent at a one-tenth price. The difference is the agent re-reading what it can no longer see.
A second analogy: masking is like clearing your desk of every printout older than an hour. The desk is tidier, but if you keep walking back to the printer for the same pages, you lose the time you hoped to save.
Compaction is still useful without a cache discount or when a context window is about to overflow. It belongs in your context engineering plan as a priced decision, not a default.
Adding context: retrieved code in the prompt
The evidence package in the paper is built by a dense code-search model that ranks functions from the issue text. The top three non-test functions appear in full, seven more are listed by location, and a code-graph retriever adds related code up to about 1,000 lines, roughly 14,000 tokens.
It worked as retrieval: searches fell, and random code of the same length changed nothing, so relevance mattered. But the package sits in every later prompt. Even as cached tokens, 14,000 extra tokens per step add up, and the saved searches did not pay for them. The authors report costs 25 to 33 percent higher even without a discount, and 80 to 127 percent higher if the package stayed until the end. Swapping the package for its short location index saved about a quarter of cost.
One result pointed the other way. Appending the retrieval ranking to the agent's final answer, which costs no prompt tokens, raised function-level localization accuracy (Acc@10) by 14 to 20 points for three models. For more on retrieval design itself, see RAG from naive to agentic.
Continue learning
How this article connects
11 pages link to or from it. Drag to turn, click a dot to open it.
How to measure agent cost correctly
The paper's methods section doubles as a measurement checklist.
Price the task, not the call. Sum uncached, cached and output tokens over the full run, and record whether the task was solved. A method that shortens each prompt but adds steps can lose.
Log the three streams separately. Providers report them: Anthropic returns cache_read_input_tokens, cache_creation_input_tokens and input_tokens; OpenAI returns usage.input_tokens_details.cached_tokens; Gemini returns usage.total_cached_tokens. With the streams stored, you can reprice any run at a new cache price later.
Report cost at several cache prices. The paper shows the cost ranking of six methods changes between a one-tenth cache price and no discount. A single price hides that.
Isolate concurrent runs. In the paper, conditions of the same task that shared a seed and ran side by side on one server reused each other's cached prefixes. That added 3.7 to 6.7 percent of prompt tokens to the cached share of the affected conditions. On Verified, the authors report the evidence package's measured cost premium was +8 percent with this sharing and +36 percent once it was removed. Running each condition alone on a fresh server gave +29 percent. Count only a run's own cache hits, or run conditions in separate processes.
Check cache behaviour per provider. Anthropic notes a cache entry becomes available only after the first response starts, so parallel requests sent at once may all miss. Gemini 4 caches the full previous request in an agent loop only if the new request adds at least 2,050 tokens.
Measure quality on the same runs. The paper uses the official SWE-bench harness. SWE-bench draws 2,294 tasks from GitHub issues in 12 Python repositories; Verified is a human-filtered subset of 500. A cost win that lowers the resolve rate is not a win. The site's guide to evals covers how to set this up.
What it means in practice
On hosted APIs, cache reads are cheap, so the cheapest history is often the one you never edit. Tricks that rewrite the middle of the prompt must clear a high bar, and tricks that add a large block to every turn must save many steps.
For teams self-hosting open models, as described in self-hosted open-weight coding agents, prefix caching saves prefill compute instead of invoice dollars, and it costs GPU memory. That is where work like the Gemma 4 K=V paper fits: storing fewer numbers per token leaves room for more cached blocks. The trade-off between context editing and cache reuse is the same; only the currency changes.
For anyone comparing agent designs, running variants in parallel on a shared server can make one look cheaper because its siblings warmed its cache. In multi-agent setups, as in subagent orchestration for coding agents, each subagent's cost likewise depends on whether its prompt starts with an already cached prefix.
What to do now
- Log cached, uncached and output tokens for every call. Store them per step and per task so you can reprice runs later.
- Order prompts from most to least stable. Tools and system instructions first, the task next, the growing history last, with timestamps at the end.
- Append instead of rewriting. Edit the middle of the history only when a measured saving justifies the rebuild.
- Make compaction price-aware. Mask or summarize in large, rare batches, or use a break-even rule that weighs savings against re-prefilling everything after the edit.
- Keep tool definitions fixed within a session. Anthropic lists tool-definition changes as invalidating the whole cache; OpenAI suggests restricting tools with
tool_choiceorallowed_toolsrather than removing them. - Shrink retrieved context after use. Swap inserted code for a short location index once the agent has found the right files.
- Count steps alongside tokens. Track steps, reads, searches and resolve rate per task.
- Isolate experiments. Run each condition in its own server process, or count only its own cache hits.
- Reprice at several cache prices. Report each design at no discount, at 0.1 and at your provider's actual rate, and fold the results into your cost and latency budgets.
Risks and open questions
The prices are reference values. The paper prices runs with chosen ratios, not real invoices, and leaves out cache-write premiums.
Cache hits are idealized. The overlap measure ignores cache expiry and eviction, and vLLM caches in fixed blocks, so real hit rates can be lower.
Open models, coding tasks. Closed frontier models and non-coding agents may react differently to missing context.
One retrieval design. The authors say their results concern this evidence package and exposure schedule; smaller or better-timed retrieval may still pay off.
Preregistration is internal. The tests were preregistered in dated internal records, not a public registry.
Prices keep moving. Cache reads already go as low as 0.025 of base input on some models, which makes cache-breaking edits relatively more expensive.
Watch next: whether agent frameworks ship cache-aware compaction by default, whether follow-up work repeats these tests on closed models and non-coding agents, and whether exact KV-cache layouts like the Gemma 4 K=V design land in mainline engines.
Frequently asked questions
What is prefix caching in simple terms?
It is reusing the attention keys and values a model already computed for the start of a prompt. If a new request begins with exactly the same tokens, the server skips recomputing that part and processes only the new tail. Providers then bill those reused tokens at a lower rate.
Why does editing an old message break the cache for everything after it?
Each token's cached keys and values depend on all tokens before it. If you change a token early in the prompt, every later token's cache is no longer valid. The server must recompute from the first changed token to the end.
Does this mean context compaction is a bad idea?
No. Without a cache discount, the paper found masking saved 19 to 36 percent, and compaction is still needed when a context window would overflow. The lesson is to price it: compact rarely, in batches, and only when the expected saving exceeds the cost of rebuilding the cache.
Should I stop adding retrieved code to agent prompts?
Not necessarily. In this study the retrieved package cut searches but raised total cost and did not measurably improve resolution. Smaller packages, removing the package after it is used, or using retrieval as a final check may work better, and the authors say other designs may still help.
How do I check my own cache hit rate?
Read the usage fields your provider returns on every call and sum them per task. Divide cached input by total input, and track it next to steps and resolve rate.
Summary
A 17 November 2026 study of more than 20,000 coding-agent runs shows that context is not free to add or remove. Under prefix caching, a history that grows only at the end is cheap. Retrieved code in every turn and masking that edits the middle both raised cost when cached input cost a tenth of uncached input, and agents often took more steps when information was missing. Keep prompts append-only by default, make compaction price-aware, log the three token streams, measure cost per task with steps and quality, and isolate experiments.
Sources
- Paper arXiv: Context Has a Price: Adding and Removing Context for LLM Coding Agents under Prefix Caching
- Paper arXiv HTML: Context Has a Price, full text
- Paper arXiv: Storing the Gemma 4 K=V Cache Once
- Web Anthropic: Prompt caching
- Docs OpenAI: Prompt caching
- Official Google: Gemini API context caching
- Docs vLLM: Automatic prefix caching
- Docs vLLM: Prefix caching design
- Web SWE-bench: Leaderboards and datasets
Cite this article
Kelahmet, Ö. C. (2026, November 19). Context has a price: why trimming a coding agent's prompt can double its bill. Özer Cem KELAHMET. https://www.ozercemkelahmet.com/articles/context-price-coding-agents-prefix-caching
BibTeX
@online{kelahmet2026context,
author = {Kelahmet, {\"O}zer Cem},
title = {{Context has a price: why trimming a coding agent's prompt can double its bill}},
year = {2026},
month = {nov},
url = {https://www.ozercemkelahmet.com/articles/context-price-coding-agents-prefix-caching}
}Version historyfirst version
- Published
Factual corrections are listed here with what changed. Wording and formatting edits are not listed.