Preference data governance outlives training methods

An explainer making the rounds contrasts the two ways models learn to be helpful: RLHF, the InstructGPT-era pipeline that trains a reward model from human preferences and then optimizes against it with reinforcement learning, and DPO, the 2023 method that collapses that whole loop into a single classification-style objective — no separate reward model, no RL. DPO is a genuinely elegant simplification, and as more enterprises consider customizing model behavior on their own expert data, it lowers the bar to doing so. Worth understanding, and worth linking.

But the simplification is being read the wrong way, and the wrong reading is about to produce a lot of unaccountable internal models. DPO makes the optimization simpler. It does nothing to make the preference pairs neutral. Both methods learn the same thing from the same place: whatever humans preferred, in the data you collected. The algorithm changed; the governance question did not move an inch.

Here is that question, stated the way it will actually bite. Six months after a team fine-tunes an internal assistant on "expert preferences," a model version ships that behaves differently — more cautious here, more assertive there — and someone in a review asks: whose judgment became policy? If the answer is a shrug, you have automated an opinion you can no longer name, defend, or reproduce. The elegance of the training method is irrelevant to that failure; the method was never the thing you needed to remember.

What you needed to record, and what neither RLHF nor DPO records for you, is the governance around the data. Provenance: who supplied each preference, and were they representative of the users the model serves? Reviewer segmentation: did three senior people's taste quietly become the house style for everyone? Disagreement records: when annotators split on a pair, how was it resolved, and by whom — because the resolution is the policy. Policy versioning: which behavioral standard was in force for this batch of data. And regression tests: the fixtures that catch when a new preference set silently shifts behavior you'd already accepted.

None of that is optional, and DPO's convenience makes it more urgent, not less — because the easier the training loop, the more teams will run it, and the more unaccountable models will accumulate in the enterprise. The demos in these explainers — Zephyr and friends — win benchmarks like MT-Bench, and I'd add the standard caveat: a benchmark win, judged by a model or a generic panel, does not predict safety, usefulness, or cultural fit inside your specific domain. Your assistant's job is to be right for your users, not to top a leaderboard someone else defined.

Steal this before your next fine-tune, RLHF or DPO alike: treat the preference dataset as the governed artifact, not the trained weights. For each batch, record provenance, reviewer segments, disagreement resolutions, and the policy version, and attach a regression fixture set — filed exactly where your eval records live. Then the day someone asks "whose judgment is this and why did behavior change," you answer from a file instead of a shrug. The weights are downstream of the data; govern the data.

DPO simplified the recipe, not the ingredients — you are still training a model on somebody's opinion, so write down whose, before it quietly becomes your policy.