← back to the archiveCover illustration for “Regression evals: catching AI changes that break things”
TERM7mo ago·Published · Updated ·by Andy Padia

Regression evals: catching AI changes that break things

In short: Regression evals rerun a fixed test set whenever a prompt, model or tool changes, so a quality drop blocks the release before users ever see it.

Regression evals are evaluations rerun automatically on every change to an AI system, such as a new prompt, model version, retrieval setting or tool, to catch anything the change breaks that used to work. OpenAI's evaluation best practices, read in November 2026, call the habit continuous evaluation: run evals on every change. They are the AI equivalent of a regression test suite in CI.

Regression evals in plain words

Before repainting one room, you check that the other rooms still have their doors on.

How it works

  1. Keep a fixed, versioned eval set, including past failures.
  2. Run it in CI for every change that touches prompts, models or retrieval.
  3. Compare per-case and aggregate scores with the last accepted version.
  4. Fail the build, or require sign-off, when key metrics fall.

Because model outputs vary, run important cases more than once and give each metric a tolerance band. Promptfoo's CI/CD guide shows the gate in practice: its example workflow runs on pull requests that touch the prompts folder, and its example script fails the build when the pass rate drops below 95 percent.

Example

# promptfoo-style config, as an illustration
prompts:
  - file://prompts/refund_v7.txt
providers:
  - anthropic:messages:<model>
tests: file://tests/refund_golden.yaml
defaultTest:
  assert:
    - type: llm-rubric
      value: "Follows refund policy; never promises amounts above the limit"

The keys follow promptfoo's configuration guide. Run it in CI and let a drop below the agreed pass rate block the merge.

Where you'll meet it

  • CI pipelines for AI features
  • Model upgrade projects when a provider releases a new version
  • Prompt management tools with versioning and comparisons
  • Vendor migrations between model providers

What it changes for you

For developers, regression evals bring prompt changes closer to the safety of code changes. For tech leads, they turn a model upgrade from a leap of faith into a measured migration. For product teams, they catch the failure nobody notices at first: a new model that is better on average but worse on your most important customer cases.

Regression evals vs online evaluation

Regression evals run before release on a fixed set. Online evaluation scores live traffic after release. You need both: one prevents known breaks, the other finds new ones.

Common mistake

My view: only watching the average score. A change can raise the average while breaking every case in one category. Report scores per category and treat any category drop as a stop.

Evals · Golden dataset · Online evaluation · LLMOps · LLM-as-a-judge

From the archive: Coding-agent vendor choice needs a regression test · Version control needs eval history

Report regression scores per category, and treat any category drop as a stop.

#field-guide#evaluation#evals#testing#ci-cd

Sources

  1. promptfoo.dev/docs/integrations/ci-cd
  2. developers.openai.com/api/docs/guides/evaluation-best-practices
  3. promptfoo.dev/docs/configuration/guide
← older drop
Fine-tuning: adapting a trained model with your examples
newer drop →
Online evaluation: scoring AI quality on live traffic

related drops

explore all 557 drops →
← back to the archiveday 175