
LLMOps: running LLM applications in production
In short: LLMOps is the practice of deploying, monitoring, evaluating and improving LLM applications in production, a specialised branch of MLOps.
LLMOps, short for large language model operations, is the set of practices and tools for running LLM applications in production: versioning prompts and models, evaluating quality, tracing requests, monitoring cost and latency, handling incidents and improving the system from real usage. Google Cloud's explainer, read in November 2026, calls it a specialised subset of MLOps.
LLMOps in plain words
It is DevOps for AI features: everything needed so the clever demo keeps working, affordably, on a Tuesday night with real users.
How it works
A typical LLMOps loop:
- Version prompts, model choices, retrieval settings and tools together.
- Evaluate changes with regression evals before release.
- Deploy through gateways with routing, limits and fallbacks.
- Observe traces, cost, latency and quality scores in production.
- Improve by turning failures into eval cases and updates.
OpenAI's evaluation guide calls step 2 continuous evaluation: run evals on every change. For step 4, OpenTelemetry's generative AI conventions define shared span, metric and event names for model calls.
Example
As an illustration, a provider releases a new model version. The team runs its regression suite on the new version, sees a small gain overall but a drop on refund questions, adjusts the prompt, reruns, then shifts traffic gradually through the gateway while watching online quality scores.
Where you'll meet it
- Observability and evaluation platforms
- Prompt management and versioning tools
- AI gateways and model registries
- Telemetry standards such as OpenTelemetry's generative AI conventions
What it changes for you
For engineering teams, AI features need the same operational discipline as payments or search: owners, dashboards, alerts and runbooks. My view: for leaders, operational maturity decides whether a pilot becomes a reliable product more often than model choice does.
LLMOps vs MLOps
MLOps covers the general practice of managing machine learning models. Google Cloud describes LLMOps as the subset that deals with the large size, complex training requirements and high computational demands of LLMs. My view: for teams that consume models through an API, the daily work is prompts, retrieval, tools and evals, with little training.
Common mistake
My view: treating prompts as configuration nobody reviews. A one-line prompt change can alter behaviour for every user. Put prompts in version control and run evals on every change.
Related terms
AI observability · LLM tracing · Regression evals · AI gateway · Online evaluation
From the archive: Version control needs eval history · Agent loops are release artifacts
Put prompts in version control and run evals on every change.


