
AI observability: seeing what your AI system actually did
In short: AI observability means capturing traces, inputs, outputs, tool calls, cost, latency and quality for each AI request so you can debug and improve it.
AI observability is the ability to see what your AI system actually did on each request: the prompt and model used, the context retrieved, the tools called, the output, the latency, the cost and the quality of the result. It extends ordinary logs, metrics and traces with AI-specific data. OpenTelemetry's GenAI semantic conventions, read in November 2026, define spans, metrics and events for it.
AI observability in plain words
It is a flight recorder for every AI conversation, so when something goes wrong you can replay exactly what happened instead of guessing.
How it works
Instrumentation captures each model call and tool call as a span in a trace. The OpenTelemetry span conventions, still marked Development, name operations such as chat, execute_tool and invoke_agent, so a whole agent run reads as one tree. Dashboards aggregate cost, latency and error rates, evaluation jobs attach quality scores, and alerts fire on drops.
Payloads need care. The conventions say instrumentations should not capture message content by default and should offer an opt-in, because that content is likely to contain sensitive user data.
A shared format also keeps the data portable. Langfuse, an open-source platform, says it builds on OpenTelemetry to reduce vendor lock-in.
Example
As an illustration, a user complains the assistant gave a wrong refund amount. The trace shows the retrieval step returned last year's policy because the new one lacked a metadata tag. The fix takes an hour, where guessing at the prompt could have taken a week.
Where you'll meet it
- Platforms such as Langfuse, LangSmith, Arize Phoenix and Braintrust
- OpenTelemetry instrumentation libraries
- AI gateways that log every call
- Incident reviews for AI features
What it changes for you
For developers, observability turns non-deterministic failures into debuggable ones. For product teams, it shows what users actually ask, which is the best source of new features and eval cases. For privacy teams, it means storing prompts and outputs that may contain personal data, so redaction and retention must be designed in.
AI observability vs traditional monitoring
Traditional monitoring asks "is it up and fast?". AI observability also asks "was the answer right, grounded and safe, and why not?".
Common mistake
My view: logging only model calls. In agents, failures often live in the steps around the model: retrieval, tool arguments and tool errors. Trace the whole run.
Related terms
LLM tracing · Online evaluation · LLMOps · AI gateway · Cost per task
From the archive: Local AI-agent observability needs a rollback path
Trace the whole agent run, with retrieval, tool arguments and tool errors, and never the model calls alone.


