
LLM tracing: following one AI request step by step
In short: LLM tracing records every step of an AI request, including model calls, retrieval and tool calls, as linked spans you can debug and cost.
LLM tracing is recording each step of an AI request or agent run as a linked set of spans, called a trace, so you can follow the path from user input to final answer: every model call, retrieval and tool call, with timing, token counts and errors. The OpenTelemetry generative AI conventions, still marked Development in November 2026, define standard names for these spans.
LLM tracing in plain words
A detailed itinerary of a journey: every stop, how long it took and what happened there, instead of only "left at 9, arrived at 11".
How it works
A span represents one unit of work, and a trace is the set of related spans that shows the path of one request, as the OpenTelemetry traces documentation describes it. Instrumentation, from SDK wrappers, frameworks or a gateway, opens a span for each operation and nests child spans under their parent.
Spans carry attributes. The generative AI conventions define names such as gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, and cover model calls, retrieval, tool execution and MCP. Traces are sent to a backend where you can search, compare and score them.
Example
As an illustration, with made-up timings and cost:
trace: "refund for order 8812" 4.8s ₹1.9
├─ llm: plan 1.1s
├─ tool: get_order(8812) 0.3s ok
├─ retrieval: refund policy (3 chunks) 0.4s
├─ llm: decide 1.6s
└─ tool: issue_refund(8812, 1499) 1.4s ERROR: amount exceeds limit
The failing step is obvious, and so is where the time and money went.
Where you'll meet it
- AI observability platforms and agent frameworks with built-in tracing
- OpenTelemetry collectors feeding existing monitoring tools
- Coding-agent session logs
- Evaluation tools that score individual traces
What it changes for you
For developers, tracing turns "the agent did something weird" into a specific span to fix. For finance, per-trace cost shows which features and users drive spend. For quality teams, traces are the raw material for online evaluation and new eval cases.
LLM tracing vs logging
Logs are separate events. Traces link events into one request's story, with parent-child structure and timing.
Common mistake
My view: tracing in development and turning it off in production to save cost. Production is where the surprising failures happen. Sample traces in production rather than dropping them entirely.
Related terms
AI observability · Agent loop · Online evaluation · LLMOps · Cost per task
From the archive: Local AI-agent observability needs a rollback path
Keep tracing on in production with sampling, because production is where the surprising failures happen.


