← back to the archiveCover illustration for “LLMOps: running LLM applications in production”
TERM7mo ago·Published · Updated ·by Andy Padia

LLMOps: running LLM applications in production

In short: LLMOps is the practice of deploying, monitoring, evaluating and improving LLM applications in production, a specialised branch of MLOps.

LLMOps, short for large language model operations, is the set of practices and tools for running LLM applications in production: versioning prompts and models, evaluating quality, tracing requests, monitoring cost and latency, handling incidents and improving the system from real usage. Google Cloud's explainer, read in November 2026, calls it a specialised subset of MLOps.

LLMOps in plain words

It is DevOps for AI features: everything needed so the clever demo keeps working, affordably, on a Tuesday night with real users.

How it works

A typical LLMOps loop:

  1. Version prompts, model choices, retrieval settings and tools together.
  2. Evaluate changes with regression evals before release.
  3. Deploy through gateways with routing, limits and fallbacks.
  4. Observe traces, cost, latency and quality scores in production.
  5. Improve by turning failures into eval cases and updates.

OpenAI's evaluation guide calls step 2 continuous evaluation: run evals on every change. For step 4, OpenTelemetry's generative AI conventions define shared span, metric and event names for model calls.

Example

As an illustration, a provider releases a new model version. The team runs its regression suite on the new version, sees a small gain overall but a drop on refund questions, adjusts the prompt, reruns, then shifts traffic gradually through the gateway while watching online quality scores.

Where you'll meet it

  • Observability and evaluation platforms
  • Prompt management and versioning tools
  • AI gateways and model registries
  • Telemetry standards such as OpenTelemetry's generative AI conventions

What it changes for you

For engineering teams, AI features need the same operational discipline as payments or search: owners, dashboards, alerts and runbooks. My view: for leaders, operational maturity decides whether a pilot becomes a reliable product more often than model choice does.

LLMOps vs MLOps

MLOps covers the general practice of managing machine learning models. Google Cloud describes LLMOps as the subset that deals with the large size, complex training requirements and high computational demands of LLMs. My view: for teams that consume models through an API, the daily work is prompts, retrieval, tools and evals, with little training.

Common mistake

My view: treating prompts as configuration nobody reviews. A one-line prompt change can alter behaviour for every user. Put prompts in version control and run evals on every change.

AI observability · LLM tracing · Regression evals · AI gateway · Online evaluation

From the archive: Version control needs eval history · Agent loops are release artifacts

Put prompts in version control and run evals on every change.

#field-guide#llmops#operations#evaluation#observability

Sources

  1. cloud.google.com/discover/what-is-llmops
  2. developers.openai.com/api/docs/guides/evaluation-best-practices
  3. github.com/open-telemetry/semantic-conventions-genai
← older drop
Plan mode: let the coding agent think before it edits
newer drop →
EU AI Act: Europe's risk-based law for AI systems

related drops

explore all 557 drops →
← back to the archiveday 175