← back to the archiveCover illustration for “LLM-as-a-judge: using a model to grade AI output”
TERM8mo ago·Published · Updated ·by Andy Padia

LLM-as-a-judge: using a model to grade AI output

In short: LLM-as-a-judge uses a language model with a rubric to score other AI outputs at scale, once the judge itself is checked against human labels.

LLM-as-a-judge is an evaluation method where a language model, given a rubric and sometimes a reference answer, scores or compares outputs from an AI system. It lets teams grade thousands of open-ended answers where exact-match checks fail. A June 2023 study found GPT-4 judges agreed with human preferences over 80% of the time, the same level as agreement between humans.

LLM-as-a-judge in plain words

Picture a teaching assistant with a marking scheme who grades the stack of essays, while the professor spot-checks a sample.

How it works

  1. Write a rubric with clear criteria and examples of each score.
  2. Give the judge the input, the output and any reference material.
  3. Ask for a score plus a short reason, or a choice between two outputs.
  4. Validate the judge against human labels on a sample before trusting it.

OpenAI's evaluation guide, read in November 2026, gives the same order: validate a judge's agreement against your human labels before optimising for cost or latency.

Example

Score the answer 1-5 for groundedness.
5 = every claim is supported by the provided sources.
3 = mostly supported, one unsupported detail.
1 = main claim unsupported or contradicted.
Sources: {sources}
Answer: {answer}
Return JSON: {"score": n, "reason": "..."}

Where you'll meet it

  • Evaluation frameworks and observability platforms
  • RAG quality metrics such as groundedness and relevance
  • Model comparisons and leaderboards based on pairwise preferences
  • Online monitoring of production traffic samples

What it changes for you

For developers, judges make evaluation of open-ended features affordable and fast. For teams, the judge itself becomes a component that needs evals: its prompt, model and rubric must be versioned and checked. For leaders, it scales human review and still depends on it.

LLM-as-a-judge vs human evaluation

Humans are the reference for quality but slow and costly. Judges are fast and cheap but carry biases. Calibrate the judge on human labels, then let it scale.

Common mistake

My view: using the same model family to generate and to judge without checking. The 2023 study names position, verbosity and self-enhancement biases in LLM judges. Swap positions, control length and compare against human labels.

Evals · Groundedness · Golden dataset · Online evaluation · Hallucination

From the archive: When should an LLM judge block an AI release?

Check a judge against human labels, with positions swapped and length controlled, before you trust its scores.

#field-guide#evaluation#evals#llm

Sources

  1. arxiv.org/abs/2306.05685
  2. developers.openai.com/api/docs/guides/evaluation-best-practices
← older drop
Agent sandbox: an isolated place for AI agents to act
newer drop →
Lethal trifecta: the agent setup that leaks your data

related drops

explore all 557 drops →
← back to the archiveday 175