← back to the archiveCover illustration for “Online evaluation: scoring AI quality on live traffic”
TERM7mo ago·Published · Updated ·by Andy Padia

Online evaluation: scoring AI quality on live traffic

In short: Online evaluation scores the real production traffic of a deployed AI system with automated graders and user signals, catching failures offline tests miss.

Online evaluation is the practice of continuously scoring the quality of a deployed AI system on live traffic, using automated graders, user feedback and business outcomes. It complements offline tests on a fixed set and catches what they miss: new kinds of questions, drifting content and provider model changes. LangSmith's documentation, read in November 2026, calls it production monitoring.

Online evaluation in plain words

Offline evals are the driving test. Online evaluation is the dashcam and the insurance record after you are on real roads.

How it works

  1. Log inputs, retrieved context, tool calls and outputs with traces.
  2. Sample a share of traffic and run graders: groundedness, policy compliance, format checks.
  3. Combine with signals such as thumbs up or down, edits, escalations, retries and task completion.
  4. Alert on drops, review samples, and feed failures back into the golden dataset.

LangSmith splits the work in two: offline evaluations for pre-deployment testing, online evaluations for production monitoring. The shared vocabulary is still settling. OpenTelemetry's generative AI conventions define a gen_ai.evaluation.result event that was still marked as in development in November 2026.

Example

As an illustration, a support assistant scores a random 5% of conversations nightly for groundedness and policy compliance. One week, groundedness falls on billing questions. Investigation shows a pricing page changed but the index was not refreshed. Offline evals, built on old content, never saw it.

Where you'll meet it

  • Evaluation and observability platforms such as LangSmith
  • Product analytics combined with AI traces
  • A/B tests comparing prompts or models on live users
  • Telemetry standards such as OpenTelemetry's generative AI conventions

What it changes for you

For developers, production becomes the most honest test set you have; OpenAI's evaluation guide lists eval datasets that do not reproduce production traffic patterns as an anti-pattern. For product owners, quality becomes a dashboard metric alongside latency and cost. For privacy teams, logging AI traffic means personal data, so retention, redaction and consent need deciding before you turn it on.

Online evaluation vs offline evaluation

Offline evaluation runs a fixed test set before release. Online evaluation measures real usage after release. Each catches what the other cannot.

Common mistake

My view: relying on thumbs-down rates alone. Few users click, and I would not assume the ones who do are representative. Pair user signals with automated graders on a random sample.

Evals · AI observability · LLM tracing · Regression evals · Groundedness

From the archive: How to evaluate an LLM feature before production · Monitor the cohort that justified the model

Pair user signals with automated graders on a random sample of live traffic.

#field-guide#evaluation#observability#llmops

Sources

  1. docs.langchain.com/langsmith/evaluation-concepts
  2. developers.openai.com/api/docs/guides/evaluation-best-practices
  3. github.com/open-telemetry/semantic-conventions-genai
← older drop
Regression evals: catching AI changes that break things
newer drop →
Small language model (SLM): compact AI for narrow jobs

related drops

explore all 557 drops →
← back to the archiveday 175