# Cost and latency budgets

Cost and latency budgets cap how much time, tokens and money an AI agent may spend per task. Learn how to set budgets, enforce them and cut cost without losing quality.

## Overview

Every agent run consumes time, tokens and money. A simple chat answer may cost a fraction of a cent and take two seconds. An agent that searches, reads documents, calls tools and checks its work may make dozens of model calls, process hundreds of thousands of tokens and run for minutes. Because agents choose their own steps, the same kind of request can cost ten times more on one run than another.

Cost and latency budgets set explicit limits on that spending: per task, per user, per team and per month. The system tracks consumption as the run proceeds and enforces the limits. When a budget is close to running out, it changes behaviour on purpose: it switches to a cheaper model, stops exploring and summarises what it has, or asks a person to approve more. Without budgets, a confused agent can loop for minutes and spend far more than the task is worth.

A household budget is a simple comparison. You decide how much to spend on groceries, fuel and entertainment, track spending as you go, and adjust before the month ends instead of discovering the overspend when the bank statement arrives. Agents need the same discipline, measured in steps, tokens, seconds and money.

## What it is

Budgets come in several forms, and good systems use more than one.

| Budget | What it limits | Typical example |
|---|---|---|
| Step budget | Model or tool calls per task | At most 15 agent steps |
| Token budget | Input and output tokens per task | At most 200,000 input tokens |
| Time budget | Wall-clock time before returning or escalating | Answer within 60 seconds |
| Money budget | Spend per task, user, team or month | 0.20 per task, 5,000 per month for a team |
| Latency target | Expected response speed | First token within 2 seconds at the 95th percentile |
| Concurrency limit | Parallel runs or sub-agents | At most 5 sub-agents at once |
| Tool budget | Calls to expensive or rate-limited tools | At most 3 paid data lookups per task |

Two kinds of latency matter. **Time to first token** is how long the user waits before anything appears. **Total time** is how long until the task is finished. Streaming can improve the first without changing the second.

## Where it came from

Budgets for computing resources are old: time limits on batch jobs, query timeouts in databases, quotas in cloud accounts and latency targets in service-level objectives. Large language models made the unit of cost the token, and providers price input and output tokens separately, with output usually costing more.

As applications moved from single prompts to agents in 2024 and 2025, cost per task became less predictable and harder to explain. Providers introduced features that change the economics. Prompt caching lets repeated prompt prefixes, such as long system prompts or documents, be reused at a discount; on Anthropic's API, for example, reading from the cache costs a tenth of the normal input price, while writing to it costs somewhat more. Batch interfaces from major providers process non-urgent requests asynchronously at a lower price. Smaller, faster models made routing simple steps away from large models practical.

At the same time, agent frameworks and coding agents added limits such as maximum turns per run, and observability tools began reporting cost per trace. Budgets became a standard part of running agents in production.

## How it works

### 1. Measure first

Use [tracing](/tools/tracing) to record tokens, model, latency and estimated cost for every step and run. Group by task type. Look at the distribution, not just the average: a small share of long runs often drives most of the cost.

### 2. Set limits per task type

From the measurements, choose limits that cover legitimate hard cases with some headroom, for example the 95th percentile of good runs plus a margin. Different task types get different budgets.

### 3. Track during the run

The runtime keeps counters for steps, tokens, time and estimated money. Before each step it checks what remains.

### 4. Degrade gracefully

When a budget is nearly spent, the agent changes mode rather than failing silently. Options include:

- switch to a cheaper or faster model for the remaining steps;
- stop exploring and write the best answer from what it has, saying what is incomplete;
- return a partial result with a task handle so work can continue later; see [long-running tasks](/tools/long-running-tasks);
- ask a person to approve more budget.

### 5. Enforce hard ceilings

Outside the agent, hard limits protect the account: per-user and per-team quotas, provider spending caps and a [circuit breaker](/tools/circuit-breaker) that stops calls when error rates or costs spike.

### 6. Reduce cost per task

Common levers, each to be checked with [evals](/tools/evals):

- **Routing:** send simple steps, such as classification, to [small language models](/tools/small-language-models).
- **Caching:** keep stable content, such as system prompts and tool definitions, at the start of the prompt so it can be cached.
- **Context trimming:** pass summaries and relevant excerpts instead of whole documents; see [context engineering](/tools/context-engineering).
- **Compact tool results:** ask tools to return the fields needed, not full pages.
- **Parallelism:** run independent calls at the same time to cut wall-clock time.
- **Batching:** use batch interfaces for work that can wait.
- **Serving optimisations:** for self-hosted models, techniques such as [speculative decoding](/tools/speculative-decoding) and efficient servers like [vLLM](/tools/vllm).

## Key concepts and terms

| Term | Meaning |
|---|---|
| Token | The unit of text a model reads or writes, and the unit of price |
| Cost per task | Total spend for one completed task, across all calls |
| Step limit | The maximum number of agent loop iterations |
| Time to first token | Delay before the first output appears |
| Tail latency | Response time at a high percentile, such as the 95th |
| Prompt caching | Reusing a processed prompt prefix at reduced cost |
| Model routing | Choosing a model per request or step by difficulty |
| Graceful degradation | Reducing scope deliberately when limits are near |
| Hard ceiling | A limit that cannot be exceeded, such as an account cap |
| Unit economics | Revenue and cost per task or per user |
| Batch processing | Non-urgent requests processed later at a lower price |

## A worked example

This is an illustrative scenario. A research agent answers analysts' questions using web search and internal documents. Finance flags that its monthly cost has tripled.

1. **Measure.** Traces from 2,000 runs show a median of 9 steps, but 6 percent of runs take more than 30 steps. Those long runs account for almost 40 percent of spend. Most of them repeat similar searches.
2. **Budget.** The team sets a 15-step limit and a 90-second time budget. At step 12 the agent is told to stop searching and write its answer, listing open questions.
3. **Route.** The first step, which classifies the question type, moves from the large model to a small one.
4. **Cache.** The 6,000-token system prompt and tool definitions are moved to a stable prefix and cached.
5. **Compact tools.** The fetch tool now returns the relevant paragraphs and title, not the whole page, cutting input tokens per step by about half.
6. **Check quality.** The team re-runs its 150-question eval set. Answer quality scores stay level, within a point of the baseline.
7. **Result.** Average cost per task falls by more than half, median latency drops from 48 to 31 seconds, and the budget cut-off triggers on about 4 percent of runs, each of which returns a partial answer with clear open questions.

## Strengths and limitations

**Strengths:**

- Predictable costs and clear unit economics.
- Protection against loops and runaway runs.
- Better user experience through consistent response times.
- Visible trade-offs between quality, speed and price.
- A structured place for human approval of expensive work.

**Limitations:**

- Limits that are too tight cut off legitimate hard tasks.
- Cost estimates during a run are approximate.
- Optimisations such as routing and trimming can lower quality if not tested.
- Different task types need different budgets, which adds configuration.
- Prices and model options change, so budgets need regular review.

## How it compares with alternatives

| Approach | Effect | Gap |
|---|---|---|
| No limits | Maximum freedom for the agent | Unpredictable cost and latency |
| Provider spending caps | Protect the account as a whole | Do not stop one runaway task early |
| Per-task budgets | Fine-grained control and graceful degradation | Need measurement and tuning |
| Model routing and caching | Lower cost per task | Do not cap worst cases on their own |
| Rate limits and quotas | Fair sharing between users | Do not shape agent behaviour |

The strongest set-up combines per-task budgets with routing, caching and account-level ceilings.

## Practical guidance and best practices

1. Measure cost and latency per task type from traces before setting limits.
2. Set step, token, time and money budgets per task type, with headroom for hard cases.
3. Warn the agent before the limit so it can finish cleanly instead of being cut off.
4. Degrade gracefully: return the best partial answer and say what is missing.
5. Route easy steps to smaller models and keep stable prompt prefixes cacheable.
6. Ask tools for compact results and trim context to what is needed.
7. Use batch processing for work that does not need an immediate answer.
8. Check every optimisation with evals so quality does not slip unnoticed.
9. Alert on cost per task and tail latency, not just monthly totals.
10. Review budgets when models, prices or task mix change.

## Common mistakes

- Setting budgets from averages, which hides the long, expensive runs.
- Cutting runs off silently, leaving users with nothing.
- Putting changing content at the start of the prompt, which defeats caching.
- Passing whole documents and raw tool output into every step.
- Switching to a cheaper model without re-running evals.
- Relying only on a monthly account cap to control per-task cost.

## Frequently asked questions

### Why do AI agents need budgets?

Because agents decide their own steps. Without limits, loops, repeated searches and over-exploration make cost and latency unpredictable, and a single confused run can cost many times more than the task is worth.

### How can I reduce agent cost without hurting quality?

Route simple steps to smaller models, cache stable prompt prefixes, trim context, return compact tool results, run independent calls in parallel and batch non-urgent work. Check each change with an eval set to confirm quality holds.

### What should happen when a budget runs out?

The agent should return its best partial result with a clear note of what is incomplete, or ask a person to approve more budget. It should not stop silently or keep going past the limit.

### What is the difference between latency and time to first token?

Time to first token is how long the user waits before output starts appearing. Total latency is how long the whole task takes. Streaming improves the first; fewer steps, smaller models and parallel calls improve the second.

### How does prompt caching save money?

When the start of a prompt is identical across calls, the provider can reuse its processed form. Cached reads are billed at a reduced rate; on some APIs, such as Anthropic's, at a tenth of the normal input price. Keeping system prompts and tool definitions stable and first makes caching effective.

### How do I choose the right step limit?

Look at traces of successful runs for each task type and set the limit above the range that good runs need, often around a high percentile plus a margin. Then watch how often the limit triggers and whether those runs were truly stuck.

## Glossary

- Budget: a limit on steps, tokens, time or money.
- Token: a unit of model input or output.
- Step: one iteration of the agent loop.
- Routing: choosing a model by task difficulty.
- Prompt cache: stored processing of a repeated prompt prefix.
- Batch interface: delayed processing at a lower price.
- Tail latency: slow responses at high percentiles.
- Graceful degradation: deliberate reduction of scope near a limit.
- Spending cap: a hard account-level limit.
- Unit economics: cost and value per task.

## Summary

Cost and latency budgets make agents predictable. Measure steps, tokens, time and money per task type from traces, set limits with headroom, track them during each run and degrade gracefully when they run low, with hard ceilings at account level as a backstop. Combine budgets with routing to smaller models, prompt caching, compact context and tool results, parallel calls and batch processing, and confirm every saving with evals so quality does not slip.