A correct rejection can still send the rewrite in the wrong direction
Netflix evaluates why its LLM judge rejects an explanation because that reason drives revision. The next useful metric is whether the prescribed repair actually fixes the defect.
Netflix's August paper on LLM judges describes a critic with two jobs. It rejects unsuitable recommendation explanations, then supplies a reason that the generator uses to revise them. A correct rejection with an incorrect reason can therefore direct the next attempt toward the wrong repair. The production lifecycle.
The paper measures agreement about reasons as well as agreement about pass or fail. That is the right distinction. I would take the next step and measure whether following the reason actually removes the intended defect without damaging something that was already correct.
Once criticism drives an automated rewrite, the criticism is part of the program's behaviour. A judge can look accurate on a spreadsheet while giving poor instructions to the component that acts on its output.
A shared fail label can hide two diagnoses
Consider a hypothetical support answer with an incorrect delivery date and an unnecessarily sharp tone. The human reviewer rejects it because the promised date is unsupported. The model judge also rejects it, but complains only about tone. Label agreement is perfect for that example; the diagnoses disagree.
Feed the judge's reason back to the writer and it may produce a warmer version of the same false promise. I would score that revision as a failed repair even if another style-oriented judge prefers it. The actionable defect survived the entire loop.
Netflix's paper explicitly targets cases where a human and a judge both reject an output but disagree about why. It reports 98.6% agreement with human judgments on 300 rationale pairs for its rationale meta-judge. That figure describes this narrower comparison, not the accuracy of every production approval. Rationale validation.
The distinction prevents a second measurement error. A high score for the instrument that compares explanations cannot automatically certify the full generation, judgment and revision system. Each stage changes what the next stage receives.
Test the repair, not just the critic
For a first implementation, I would collect a small set of rejected answers with independently identified defects. This is a proposed test, not an evaluation I have run on Netflix's system. Preserve the original answer, the critic's diagnosis, the revised answer and the human assessment of what changed.
The useful comparison asks three things: did the targeted problem disappear, did unrelated correct content remain intact, and did the revision introduce a new problem? A single improved quality score can conceal a trade between those outcomes.
I would include deliberately ambiguous cases too. If the critic lacks the information needed to tell whether a date is correct, it should request that evidence or identify the uncertainty. Rewriting until the prose sounds more confident is not a repair.
Retry limits belong in the same test. Netflix describes a bounded revision loop with a budget of three retries, dropping explanations that still fail. The right budget for a support answer may differ, but the system needs a terminal action other than asking the same pair of models to agree eventually. Deployment design.
The paper withholds some production criteria and model details, so its exact results are not a recipe another team can reproduce from the article alone. Its structural lesson is transferable: evaluate a judge according to the work its output causes.
Steal the support-answer fixture before adding an automatic rewrite step. If the loop fixes the tone while preserving the false date, you have found a control error that pass/fail accuracy would miss.
A useful critic must diagnose the defect well enough that acting on its advice produces a real repair.
