Running agents in production: evaluation, tracing, guardrails and cost
An agent that works in a demo fails in production in new ways: wrong tools, endless loops, injected instructions and surprise bills. This page lists the practices that keep agents reliable.
Overview
Production agents need the same discipline as any service plus a few new controls, because the agent chooses its own actions. The practices below are grouped by when they act: before release, at every step, and after something goes wrong.
Before release: evaluation
| Check | What it measures | Tools |
|---|---|---|
| Task success | Did the agent complete the task end to end? | Evals, DeepEval |
| Tool use | Right tool, right arguments, no unnecessary calls | Trace-based checks |
| Answer quality | Correct, grounded, complete | LLM as judge, RAGAS |
| Safety | Resists injection, refuses unsafe actions | Red teaming, promptfoo |
| Cost and latency | Tokens, steps and time per task | Cost and latency budgets |
Keep a fixed evaluation set and run it on every change to a prompt, model or tool, like unit tests for behaviour. A regression board shows which change broke which case.
At every step: guardrails and limits
- Input checks: detect prompt injection and personal data before the model sees them (Presidio).
- Tool permissions: least-privilege credentials per tool, and human approval for actions that change data or spend money.
- Sandboxing: run code and browsers in isolated environments (sandboxing).
- Budgets: caps on steps, tokens, time and money per task.
- Output checks: policies applied before results reach users (guardrails, NeMo Guardrails, Guardrails AI).
Seeing what happened: tracing
A trace records every model call, tool call, input and output in one run as a tree. Without traces an agent failure cannot be explained. Store them with LangSmith, Langfuse, Phoenix or any OpenTelemetry backend, and sample production traces into the evaluation set.
When something goes wrong
Durable runtimes save state after each step, so a run can resume after a crash or be replayed from any step to reproduce a bug. Keep a kill switch per agent, alert on unusual tool use or cost, and review failed traces weekly. For long tasks see long-running tasks.
Learn it hands-on
The guardrails simulation shows layered checks, the LangGraph checkpoints simulation shows resume and replay, and the article building an agent from scratch wires tracing and evaluation into a real agent. For governance see the EU AI Act and AI governance pages.
Frequently asked questions
What is the most common production failure?
Agents that loop or call the wrong tool repeatedly. Step budgets, clear tool descriptions and trace review catch most of these.
How often should the evaluation set change?
Add every new failure you find in production, and keep old cases so fixed bugs stay fixed.
Summary
Evaluate before release, limit and check every step, trace everything, and make runs resumable. These four habits matter more than the choice of framework.
Back to the topic
This page is part of Agentic AI, which maps every concept, tool, simulation and article on the subject.