Tracing for AI agents

Overview

Tracing records what an AI agent actually did during a run. It captures each model call with its model name, token counts and timing, each tool call with its arguments and result, each handoff to another agent and each retrieval step. The steps are linked into one tree, so you can follow a single request from the moment it arrives to the final answer, and see which step caused which.

Agents are hard to debug without traces. The same question can take a different path each time. A wrong answer might come from a bad retrieval, a misread tool result, a confusing prompt or a model that chose the wrong tool. Logs scattered across services rarely show the whole story. A trace does.

An aircraft's flight recorder is a fair comparison. When something goes wrong, investigators do not guess. They replay the recorded data to see what the instruments showed, what the crew did and when. Traces are the flight recorder of an agent system. The difference is that you can read them every day, not only after an accident, to find slow steps, waste and quiet failures.

What it is

Tracing uses a simple data model borrowed from distributed systems.

On top of this model, observability tools often add evaluation scores, user feedback and cost estimates to each trace.

Where it came from

Distributed tracing grew up with large web systems. Google described its internal tracing system, Dapper, in a 2010 paper, and open-source projects such as Zipkin and Jaeger followed. Two efforts to standardise instrumentation, OpenTracing and OpenCensus, merged in 2019 to form OpenTelemetry, a Cloud Native Computing Foundation project for traces, metrics and logs.

When language-model applications became common in 2023, teams needed the same visibility for prompts, completions and tool calls. Dedicated LLM tracing tools appeared, including LangSmith, Langfuse and Phoenix, and many agent SDKs began emitting traces by default.

OpenTelemetry started a set of semantic conventions for generative AI, which give standard names to operations and attributes. Operations include chat, embeddings, execute_tool and invoke_agent, and attributes include gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. These conventions were still marked as in development in 2026, so names can change between releases. In June 2026 they moved to a dedicated GenAI semantic conventions repository.

How it works

1. Instrument

Instrumentation creates spans around the operations you care about. Many agent SDKs and model client libraries do this automatically. For your own code, you open spans by hand around important steps, such as a business decision or a call to an internal service.

2. Record the right details

Each model span typically records the operation, provider, requested and returned model, input and output tokens, latency and status. Tool spans record the tool name, arguments and a summary of the result. Under the GenAI conventions, inference spans are named by operation and model, for example chat followed by the model name. Full prompts and completions are opt-in and carried as events rather than ordinary attributes, because they are large and may hold personal data.

3. Propagate context

When the agent calls a tool server or another agent, the trace context travels with the request, usually in a header. The receiving service continues the same trace, so the tree stays connected across processes and even across organisations.

4. Redact and sample

Before export, processors can mask personal data, strip secrets and drop large payloads. High-traffic systems may keep only a sample of normal traces but every trace with an error.

5. Store, search and use

The backend lets you search by user, model, tool, error, latency or cost; view a trace as a timeline or tree; compare runs; and attach evaluation scores. Interesting traces become test cases for evals.

Key concepts and terms

A worked example

This is an illustrative scenario. Complaints rise about a support agent giving wrong delivery dates.

1. An engineer filters traces from the past week for runs that called the get_delivery_status tool and received negative user feedback. There are 214.
2. Opening a few, she sees the same pattern. The tool span returns "eta": "11/05/2026". The next model span answers "Your parcel arrives on 11 May".
3. Comparing with traces from two weeks earlier, the tool used to return "2026-11-05". A recent change in the shipping service switched the date format to month-first.
4. The trace timeline also shows that each of these runs called the tool twice, because the model asked again after an unclear first answer, which doubled the tool's latency.
5. The team changes the tool to return ISO dates with an explicit time zone and adds 20 of the failing traces to the evaluation set.
6. After deployment, traces show the tool called once per run, correct dates in answers and median latency down from 4.1 to 2.6 seconds.

Without traces, the team would have seen only a vague rise in complaints and guessed at the prompt.

Strengths and limitations

Strengths:

• Turns debugging from guesswork into inspection of real runs.
• Shows latency, token use and cost per step and per run.
• Provides an audit trail of what the agent saw and did.
• Supplies real examples for evaluation and regression tests.
• Works across services and agents with standard context propagation.

Limitations:

• Storage and processing cost, especially when full content is captured.
• Privacy and security risk if prompts or tool results contain personal data or secrets.
• Shows what happened, not whether the answer was right; that needs evaluation.
• Conventions for AI spans are still evolving, so names may change.
• Missing instrumentation in one service breaks the tree.

How it compares with alternatives

Most teams use all of these together: metrics for alerts, traces for diagnosis and evals for quality.

Practical guidance and best practices

1. Add tracing from the first prototype; retrofitting it later is harder.
2. Use OpenTelemetry-compatible instrumentation so you can change backends.
3. Record model, provider, tokens, latency, status and tool names on every relevant span.
4. Propagate one trace id across services, tool servers and agents.
5. Capture full prompts and completions only where you need them, and redact personal data and secrets.
6. Keep every trace with an error and sample the rest if volume is high.
7. Attach user feedback and evaluation scores to traces.
8. Turn interesting or failing traces into evaluation cases.
9. Build saved views for the questions you ask often, such as slowest tools or most expensive runs; see cost and latency budgets.

Common mistakes

• Logging only the final answer, with no record of the steps.
• Storing raw prompts with customer data and no access controls.
• Starting a new trace in each service, so runs cannot be followed end to end.
• Using custom attribute names everywhere, which makes dashboards hard to share.
• Collecting traces but never reading them.
• Treating a clean trace as proof that the answer was correct.

Frequently asked questions

What is agent tracing?

It is recording every step of an agent run, including model calls, tool calls, retrievals and handoffs, as timed spans linked into one trace. It lets you follow exactly what the agent did and how long each part took.

Why use OpenTelemetry for AI tracing?

OpenTelemetry is a widely supported open standard for telemetry. Its generative AI conventions give common names for operations such as chat and execute_tool and attributes such as token counts, so traces can move between tools and backends. The conventions were still in development in 2026.

Do traces contain sensitive data?

They can, especially if full prompts, documents or tool results are captured. Capture content only where needed, redact personal data and secrets before export, restrict access and set retention limits.

What is the difference between tracing and logging?

Logs are separate lines written by each component. Traces are structured, timed spans linked by a shared id and parent-child relationships, so they show the whole path of one request across components.

How do traces help evaluation?

Traces supply real inputs, outputs and trajectories. Teams score them with evaluators, pick failures and edge cases, and add them to evaluation sets so the same problems are caught before future releases.

Should I trace every request?

For low and medium volume, often yes. At high volume, keep all errors and a representative sample of the rest, and capture full content for only a subset.

Glossary

• Span: a timed step in a trace.
• Trace id: the shared identifier of one run.
• Parent span: the span that started another.
• Attribute: a detail recorded on a span.
• Event: a time-stamped note within a span.
• Exporter: the component that sends spans to a backend.
• OTLP: the OpenTelemetry export protocol.
• Sampling: storing only some traces.
• Redaction: removing sensitive values.
• GenAI conventions: standard names for AI operations and attributes.

Summary

Tracing makes agent behaviour visible step by step. Each run becomes a trace of linked spans for model calls, tool calls and handoffs, with tokens, latency, cost and status recorded. Built on ideas from distributed tracing and standardised through OpenTelemetry, whose generative AI conventions were still evolving in 2026, it turns debugging into inspection and feeds evaluation with real cases. Trace from the start, propagate one id everywhere, use standard attribute names, protect sensitive content and read your traces regularly.

Related

It belongs to Operations in my stack, under Agentic AI. Others in the same group: