What a production-ready LLM architecture needs

On 26 February 2026 Google Cloud published five lessons from moving its own go-to-market organisation past generative-AI pilots, and the first one was not about models at all: curate the data and context the application is grounded in. Its architecture guidance, last reviewed 19 November 2024, is just as unglamorous — two deployment paths, batch and online; integration tests in a production-like environment; checks on permissions, infrastructure and dependencies before release.

Read together, they describe why pilots stall. A pilot is a prompt with a demo around it. A production-ready LLM architecture is a bounded service — eight components with a clear boundary — plus a failure-response map that gives every way the thing can go wrong a trace, an alert, a fallback and a named operator. The model is one box in that diagram, and rarely the one that breaks first.

The boundary: eight components, one diagram

Authentication and tenant scope decide whose data a request may touch before anything else runs. Input validation rejects what the system was never designed to answer. Orchestration owns the control flow — a workflow for most features, an agent loop with host-enforced budgets for the few that need one. Retrieval and tools are behind their own permissions, not the user's. The model gateway is the one place model IDs, versions, timeouts, retry ceilings and spend caps live — provider-neutral, so a gateway bypass is a budgeted exception rather than a habit. Output validation checks schema, policy and grounding before the answer leaves. The trace store keeps a redacted record of every hop. Feedback and evals close the loop: Google's own lesson here is unfashionably manual — one-to-five-star ratings, focus groups, and expert reviewers who write the evaluation standards.

The operating table

An architecture without owners is a diagram. Each metric below needs a denominator and a person, or it will not be looked at the week it matters.

Metric	Denominator	Owner
p50 / p95 end-to-end latency	per request, by feature	platform
Failures and timeouts	per request, by component	platform
Task acceptance	per task, by user slice	product
Unsupported claims	per sampled answer, weekly	domain reviewer
Tokens and cost per accepted task	per accepted task	product + finance
Escalation rate	per task	operations

Denominators are the point: a 2 % failure rate per request and a 2 % failure rate per task are different incidents. Thresholds come from your product, not from a vendor default.

The last two rows are where the cost conversation actually lives, and the acceptance row is what the pre-production eval was built to predict. Keep offline quality checks separate from the delayed human labels: the first run on every change, the second arrive days later and are the truth.

The failure-response map

Write the failures down with four columns — trace, alert, fallback, operator — and the architecture starts to design itself. A model timeout: traced at the gateway, alerted on p95, falls back to a cached or smaller-model answer with a visible label, owned by platform. A retrieval miss: traced with the query and the empty result, alerted on miss rate per slice, falls back to "I can't find that" rather than a confident guess, owned by the content team. A denied tool permission: traced, alerted once, no fallback — the request is refused and the attempt is logged. An unsupported claim in a sampled answer: no alert, a weekly review, and a fix in the grounding data, owned by the domain reviewer.

Three mechanisms make the map enforceable. Versioned configuration, so a prompt or model change is a release with a rollback, which is also what makes the harness a release platform. Retry ceilings and circuit breakers at the gateway, so a bad hour at a provider degrades the feature instead of the whole product. And staged rollout — a slice of tenants first, the acceptance row watched — with a rollback that is rehearsed, not assumed.

The design exercise I would run

No deployment or load test was run for this piece; this is the exercise, labelled as such. Take a document-support prototype and list every failure it can produce — timeout, empty retrieval, hallucinated reference, permission error, malformed output, cost spike. For each, fill the four columns. Where a cell is empty, that is the next engineering task; where the operator column says "nobody", that is the reason the pilot will stall. Then run Google's two paths on purpose: the batch path for evaluation and backfill, the online path behind the gateway, each with its own integration test against production-like data and permissions.

Where this holds and where it breaks

It holds for any LLM feature that answers or acts inside a product — support, extraction, drafting, internal assistants — regardless of whether an agent is involved. It is lighter for a batch pipeline, where half the table collapses into a job log. It breaks when the organisation cannot name an operator per failure; at that point the gap is ownership, and no component in the diagram will fix it.

FAQ

Do I need all eight components for a small feature?

You need all eight decisions. For a small feature several are a few lines each — but skip output validation or the trace store and the first incident will be undiagnosable.

Where do guardrails go?

In input and output validation and at the tool router, enforced by the host. The model gateway holds budgets and timeouts; it is not where policy lives.

How do I know a pilot is ready?

When the operating table has numbers in every row, each failure in the map has a named operator, and a rollback has been rehearsed once.

What's in it for you

• An eight-component service boundary you can draw on one page and check a prototype against.
• An operating table with denominators and owners, so metrics become responsibilities.
• A four-column failure map — trace, alert, fallback, operator — that turns "production-ready" into a checklist.

A production-ready LLM architecture is eight components and a failure map with a name in every operator cell; the model is the easy part.