Version control needs eval history

The advice reaching product managers this week is "put everything you build with AI in git", with the Awesome LLM Apps repository as the on-ramp — 100-plus runnable agent and RAG templates, each advertised as three commands from working, Apache-licensed for forking. (I'll skip the newsletter's time-to-restore anecdotes and star counts, which I couldn't verify and which age by the hour.) The advice is correct, and I have given it myself: git brings diffs, authorship, review, and rollback to prompts, skills, and agent configs — artifacts that today mostly live in Notion pages and chat scrollback.

But having pushed teams to do this for a year, I can report where the advice runs out, because I have watched a team follow it perfectly and still get burned. They versioned every prompt. A regression appeared after a change; they rolled back to the previous file, exactly as the playbook says. The regression stayed. Two days of confusion later, the truth emerged: the old prompt had been "safe" under a different model version and a different retrieval index, and nobody could reconstruct which combination had actually passed testing, because the repo stored the prompt and nothing else. The rollback restored the text. It could not restore the world the text used to live in.

That is the general failure: git versions the artifact, not the evidence. A prompt diff tells you what changed. It does not tell you which model the old version was validated against, which fixture set it passed, what the scores were, what failure modes were known and accepted, who decided it was good enough, or what stored context — memory files, indexes, conversation state — the version assumes. For code, tests-in-CI carry much of that burden automatically. For AI artifacts, where behavior depends on model × prompt × data × context, the burden is heavier and almost nobody carries it.

The fix is small and boring, which is why it works: every versioned AI artifact travels with an eval record — a short structured block, in the same commit, stating model and version tested, fixture set and scores, known failures accepted, reviewer, and compatibility notes for any stored context (including migration steps when the shape of memory changes). Rollback then means restoring a validated combination, not a text file. The commit message habit that follows is the one I now insist on: no eval record, no merge — the same rule as "no tests, no merge", one abstraction level up.

This also upgrades the templates themselves. Cloning a runnable agent from a repo like this one gets you to working in an afternoon — genuinely valuable. What the clone does not include is any evidence the template behaves acceptably on your inputs. The first thing to add to a forked template is not a feature; it is five fixtures and a baseline eval record, so that every subsequent change has something to be compared against.

Steal this today: add an EVALS.md next to your most-edited prompt, backfill one honest entry for the current version, and require an entry per change from now on. One file, five fields, no tooling needed to start.

Git remembers what you changed — only an eval record remembers why it was safe; version both or roll back blind.