Record/replay keeps fast-moving AI API tests honest

The most dangerous green test in an AI backend is the one that never talks to the AI backend.

Red Hat described this failure in its OpenShift AI integration with Llama Stack: mocks kept passing while the upstream API added fields, changed response shapes and deprecated endpoints. The fix was not to delete mocks. It was to put a real, pinned server into test setup, then use record/replay fixtures for the ordinary path. That combination catches API drift without turning every pull request into a paid model call.

Why mocks miss an upstream AI API change

A mock is a copy of your expectation. It is not a copy of the server. When the server changes, the mock remains perfectly obedient, so the suite proves only that your client agrees with yesterday's assumptions.

The useful boundary is therefore two tests, not one. A normal test should be deterministic and cheap. A compatibility test should exercise the real dependency and be allowed to tell you that the contract moved. Red Hat reports running both against a real Llama Stack process and using a scheduled sentinel against stable and development releases.

Start the real dependency without Docker

The reported setup starts a pinned Llama Stack version as a uv run subprocess on localhost. The suite waits for /v1/models, seeds a vector store, and waits for asynchronous embedding completion before the first assertion. That last wait matters: an index that has not finished building looks like a retrieval bug.

Here is the shape to copy, with names simplified. It is a configuration pattern, not a local test I ran:

cmd := exec.CommandContext(ctx, "uv", "run",
    "--with", "llama-stack=="+version,
    "--with", "llama-stack-api=="+version,
    "llama", "stack", "run", configPath,
    "--port", "18321",
)
if err := cmd.Start(); err != nil { return err }
defer cmd.Process.Signal(os.Interrupt)

if err := waitFor("http://127.0.0.1:18321/v1/models", 180*time.Second); err != nil {
    return err
}

Pin the version in the same file that starts the test. An unpinned integration test only moves the surprise from production to CI.

Replay by default, record deliberately

Record/replay is valuable because the real dependency can still involve an LLM provider and asynchronous state. Red Hat's pattern carries a test identifier in X-LlamaStack-Provider-Data, allowing fixtures to stay isolated. Pull requests replay committed recordings at no model cost. A maintainer enters record mode only when the client or expected contract intentionally changes, refreshes the fixtures, and reviews the diff.

My editorial judgement is that the mode switch should be visible in the command name. make test should never quietly spend money or overwrite evidence. A separate make llamastack-record makes the expensive, state-changing action auditable in a pull request.

What this catches — and what it does not

This harness catches request and response-shape drift, missing fields, endpoint changes and workflow assertions that mocks forgot to model. It does not prove model quality, concurrency behaviour, rate-limit recovery or production data permissions. Keep those as separate tests instead of claiming that replay is a complete integration environment.

What's in it for you

• Your everyday test remains fast, deterministic and free of model calls.
• An upstream release can fail a compatibility sentinel before it fails a deployed service.
• Fixture changes become reviewable evidence instead of invisible mock edits.

Run the real dependency once, replay it cheaply every day, and make every fixture refresh earn its way into the diff.