← back to the archiveCover illustration for “LLM tracing: following one AI request step by step”
TERM7mo ago·Published · Updated ·by Andy Padia

LLM tracing: following one AI request step by step

In short: LLM tracing records every step of an AI request, including model calls, retrieval and tool calls, as linked spans you can debug and cost.

LLM tracing is recording each step of an AI request or agent run as a linked set of spans, called a trace, so you can follow the path from user input to final answer: every model call, retrieval and tool call, with timing, token counts and errors. The OpenTelemetry generative AI conventions, still marked Development in November 2026, define standard names for these spans.

LLM tracing in plain words

A detailed itinerary of a journey: every stop, how long it took and what happened there, instead of only "left at 9, arrived at 11".

How it works

A span represents one unit of work, and a trace is the set of related spans that shows the path of one request, as the OpenTelemetry traces documentation describes it. Instrumentation, from SDK wrappers, frameworks or a gateway, opens a span for each operation and nests child spans under their parent.

Spans carry attributes. The generative AI conventions define names such as gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, and cover model calls, retrieval, tool execution and MCP. Traces are sent to a backend where you can search, compare and score them.

Example

As an illustration, with made-up timings and cost:

trace: "refund for order 8812"            4.8s  ₹1.9
 ├─ llm: plan                             1.1s
 ├─ tool: get_order(8812)                 0.3s  ok
 ├─ retrieval: refund policy (3 chunks)   0.4s
 ├─ llm: decide                           1.6s
 └─ tool: issue_refund(8812, 1499)        1.4s  ERROR: amount exceeds limit

The failing step is obvious, and so is where the time and money went.

Where you'll meet it

  • AI observability platforms and agent frameworks with built-in tracing
  • OpenTelemetry collectors feeding existing monitoring tools
  • Coding-agent session logs
  • Evaluation tools that score individual traces

What it changes for you

For developers, tracing turns "the agent did something weird" into a specific span to fix. For finance, per-trace cost shows which features and users drive spend. For quality teams, traces are the raw material for online evaluation and new eval cases.

LLM tracing vs logging

Logs are separate events. Traces link events into one request's story, with parent-child structure and timing.

Common mistake

My view: tracing in development and turning it off in production to save cost. Production is where the surprising failures happen. Sample traces in production rather than dropping them entirely.

AI observability · Agent loop · Online evaluation · LLMOps · Cost per task

From the archive: Local AI-agent observability needs a rollback path

Keep tracing on in production with sampling, because production is where the surprising failures happen.

#field-guide#observability#llmops#agents#debugging

Sources

  1. github.com/open-telemetry/semantic-conventions-genai
  2. opentelemetry.io/docs/concepts/signals/traces
← older drop
Benchmark contamination: when models have seen the test
newer drop →
Agent skills: packaged know-how an agent loads on demand

related drops

explore all 557 drops →
← back to the archiveday 175