
LLM-as-a-judge: using a model to grade AI output
In short: LLM-as-a-judge uses a language model with a rubric to score other AI outputs at scale, once the judge itself is checked against human labels.
LLM-as-a-judge is an evaluation method where a language model, given a rubric and sometimes a reference answer, scores or compares outputs from an AI system. It lets teams grade thousands of open-ended answers where exact-match checks fail. A June 2023 study found GPT-4 judges agreed with human preferences over 80% of the time, the same level as agreement between humans.
LLM-as-a-judge in plain words
Picture a teaching assistant with a marking scheme who grades the stack of essays, while the professor spot-checks a sample.
How it works
- Write a rubric with clear criteria and examples of each score.
- Give the judge the input, the output and any reference material.
- Ask for a score plus a short reason, or a choice between two outputs.
- Validate the judge against human labels on a sample before trusting it.
OpenAI's evaluation guide, read in November 2026, gives the same order: validate a judge's agreement against your human labels before optimising for cost or latency.
Example
Score the answer 1-5 for groundedness.
5 = every claim is supported by the provided sources.
3 = mostly supported, one unsupported detail.
1 = main claim unsupported or contradicted.
Sources: {sources}
Answer: {answer}
Return JSON: {"score": n, "reason": "..."}
Where you'll meet it
- Evaluation frameworks and observability platforms
- RAG quality metrics such as groundedness and relevance
- Model comparisons and leaderboards based on pairwise preferences
- Online monitoring of production traffic samples
What it changes for you
For developers, judges make evaluation of open-ended features affordable and fast. For teams, the judge itself becomes a component that needs evals: its prompt, model and rubric must be versioned and checked. For leaders, it scales human review and still depends on it.
LLM-as-a-judge vs human evaluation
Humans are the reference for quality but slow and costly. Judges are fast and cheap but carry biases. Calibrate the judge on human labels, then let it scale.
Common mistake
My view: using the same model family to generate and to judge without checking. The 2023 study names position, verbosity and self-enhancement biases in LLM judges. Swap positions, control length and compare against human labels.
Related terms
Evals · Groundedness · Golden dataset · Online evaluation · Hallucination
From the archive: When should an LLM judge block an AI release?
Check a judge against human labels, with positions swapped and length controlled, before you trust its scores.


