
Regression evals: catching AI changes that break things
In short: Regression evals rerun a fixed test set whenever a prompt, model or tool changes, so a quality drop blocks the release before users ever see it.
Regression evals are evaluations rerun automatically on every change to an AI system, such as a new prompt, model version, retrieval setting or tool, to catch anything the change breaks that used to work. OpenAI's evaluation best practices, read in November 2026, call the habit continuous evaluation: run evals on every change. They are the AI equivalent of a regression test suite in CI.
Regression evals in plain words
Before repainting one room, you check that the other rooms still have their doors on.
How it works
- Keep a fixed, versioned eval set, including past failures.
- Run it in CI for every change that touches prompts, models or retrieval.
- Compare per-case and aggregate scores with the last accepted version.
- Fail the build, or require sign-off, when key metrics fall.
Because model outputs vary, run important cases more than once and give each metric a tolerance band. Promptfoo's CI/CD guide shows the gate in practice: its example workflow runs on pull requests that touch the prompts folder, and its example script fails the build when the pass rate drops below 95 percent.
Example
# promptfoo-style config, as an illustration
prompts:
- file://prompts/refund_v7.txt
providers:
- anthropic:messages:<model>
tests: file://tests/refund_golden.yaml
defaultTest:
assert:
- type: llm-rubric
value: "Follows refund policy; never promises amounts above the limit"
The keys follow promptfoo's configuration guide. Run it in CI and let a drop below the agreed pass rate block the merge.
Where you'll meet it
- CI pipelines for AI features
- Model upgrade projects when a provider releases a new version
- Prompt management tools with versioning and comparisons
- Vendor migrations between model providers
What it changes for you
For developers, regression evals bring prompt changes closer to the safety of code changes. For tech leads, they turn a model upgrade from a leap of faith into a measured migration. For product teams, they catch the failure nobody notices at first: a new model that is better on average but worse on your most important customer cases.
Regression evals vs online evaluation
Regression evals run before release on a fixed set. Online evaluation scores live traffic after release. You need both: one prevents known breaks, the other finds new ones.
Common mistake
My view: only watching the average score. A change can raise the average while breaking every case in one category. Report scores per category and treat any category drop as a stop.
Related terms
Evals · Golden dataset · Online evaluation · LLMOps · LLM-as-a-judge
From the archive: Coding-agent vendor choice needs a regression test · Version control needs eval history
Report regression scores per category, and treat any category drop as a stop.


