← back to the archiveCover illustration for “Benchmark contamination: when models have seen the test”
TERM7mo ago·Published · Updated ·by Andy Padia

Benchmark contamination: when models have seen the test

In short: Benchmark contamination means a model's training data included the benchmark's test items, so its score reflects exposure more than real ability.

Benchmark contamination is when a model has already seen the test: its training data contains a benchmark's questions, answers or close variants, so the score reflects memorisation more than ability. Sainz and co-authors described the worst case in October 2023 as a model trained on a benchmark's test split and then evaluated on that same benchmark. Contaminated scores are poor guides to real performance.

Benchmark contamination in plain words

Think of a student who saw the exam paper the night before: full marks, and you learn nothing about whether they understand the subject.

How it works

Large models train on web crawls, and benchmarks, their solutions and discussions about them are published on the web. Exposure can happen without anyone intending it, and the model then reproduces answers it has effectively seen.

Sainz and co-authors call for a community effort: automatic and semi-automatic ways to detect when a benchmark was exposed to a model, and flags on papers whose conclusions are compromised. Common checks are looking for verbatim overlap, testing on rephrased or newly written questions, and comparing results on items published before and after a model's training cutoff.

Some benchmarks are built to resist the problem. LiveCodeBench, submitted in March 2024, continuously collects new problems from LeetCode, AtCoder and CodeForces contests. The paper describes four hundred problems published between May 2023 and May 2024.

Example

As an illustration, a model scores very well on a coding benchmark's older problems but drops sharply on problems published after its training cutoff. That gap suggests some of the older score came from exposure rather than skill.

Where you'll meet it

  • Debates about model launch claims and leaderboard results
  • Live or rolling benchmarks that add new problems after model release
  • Research on detecting memorisation
  • Vendor evaluations where the vendor chose the benchmark

What it changes for you

For buyers and tech leads, public benchmark scores are a starting point; they are weak evidence for your use case. The defence is your own private eval set, built from your data, which no model has seen. For researchers and vendors, the ask from Sainz and co-authors is to measure contamination for each benchmark and say so.

Benchmark contamination vs overfitting

Overfitting is a model fitting its training data too closely. Contamination is test data leaking into training, which makes the evaluation untrustworthy as well as the model.

Common mistake

My view: choosing a model from a leaderboard table. Run your top two or three candidates on fifty of your own real cases. The ranking can change, and that ranking is the one that matters to you.

Evals · Golden dataset · SWE-bench · Pass@k

From the archive: A perfect ExploitBench score changes what the test can tell us

Pick models on your own private test cases, because a public leaderboard may be measuring exposure.

#field-guide#evaluation#benchmarks#models

Sources

  1. arxiv.org/abs/2310.18018
  2. arxiv.org/abs/2403.07974
← older drop
Model routing: sending each request to the right model
newer drop →
LLM tracing: following one AI request step by step

related drops

explore all 557 drops →
← back to the archiveday 175