Contamination & Saturation

How a benchmark gets built is the pipeline Y17 takes apart. Benchmarks also have a brutal life cycle: born, adopted as a target, optimised to saturation (everyone near full marks, all discriminating power gone), devalued, replaced by the next benchmark. The most insidious accelerant is called contamination: the benchmark's questions leak into the model's training data, so the model has not acquired the ability, it has seen the answers. The three detective stories below turn on only two questions: are the questions new, and are the questions right?

First, some macro readings on the life cycle. The Stanford AI Index 2025 counts three benchmarks released only in 2023 (MMMU, GPQA, SWE-bench) whose scores jumped by 18.8, 48.9 and 67.3 percentage points respectively within a year, with SWE-bench (real GitHub bug-fixing tasks) going from 4.4% to 71.7%. Going from "can barely do any of it" to "can do most of it" in a year is either an intelligence explosion or a ruler breaking very fast, and the evidence below says a fair amount of it is water. At the other end, older benchmarks like MMLU, GSM8K and HumanEval are broadly saturated above 88% and have lost discriminating power for frontier models. Read together, the more a ruler is treated as a target, the faster it is scrapped. This is precisely the macro fingerprint of reactivity.

Detective story one: retest on new questions. Scale AI's GSM1k commissioned a thousand entirely new problems matching GSM8K (the most-used grade-school maths bank) in style, difficulty and solution length, and had the models sit them again. The Phi and Mistral families dropped systematically by 8% to 13%, meaning roughly a tenth of their original score was memorised. Better still is the forensic step: a model's probability of reproducing a GSM8K problem verbatim correlated with the size of its drop (Spearman r² of about 0.36), so the more a model looked like it had memorised the originals, the harder it fell on new ones. Converting that r² into a share, memorisation explains well over a third of the drop. The same year brought a deliberately absurd satirical paper: Schaeffer trained a one-million-parameter miniature model called phi-CTNL (read it aloud: "fictional") that scored perfectly on a pile of academic benchmarks, with exactly one trick, namely that the training data was the benchmarks' test sets themselves. The target was the launch-event culture of declaring SOTA without any contamination check. Apple's GSM-Symbolic pushed the incision one layer deeper: leave the original problems alone and just swap the numbers, and most models slid significantly. So what has been memorised is not only the problem text but the problem template itself.

Detective story two: audit the "gold answers" themselves. The other half of saturation lies not in the models but in the questions. MMLU-Redux had 5700 MMLU items relabelled by hand and estimated that 6.49% of the questions are themselves wrong: mislabelled answers, truncated stems, more than one correct option. What does that share imply? If 6.49% of the questions are wrong, then even a model that correctly answers everything it should know can score at most 100% minus that share. That is exactly why the physical ceiling on accuracy is nailed at around 93.5%: a model that knows everything it should know still tops out there. Seen from that angle, "capability saturation" is at least partly a collision with an error ceiling. Cleaning the bad items also changes model rankings, meaning the original leaderboard was partly stirred by error noise. This is not unique to MMLU: MMLU-Pro, billed as carefully refined, still shows about 4% erroneous items plus another 5% or so of generation, annotation and answer-extraction problems. A Text-to-SQL audit found that 66.1% of the gold answers across 121 open-source items were mislabelled, so two thirds of the "gold answers" were themselves wrong. Broken benchmarks are a systematic cross-domain phenomenon, and "harder" does not mean "cleaner".

The popular narrative that "saturation is accelerating" also needs discounting. Akhtar, Reuel et al. systematically analysed 60 mainstream benchmarks in 2026: the saturated fraction rises from 42.9% for benchmarks under 24 months old to 54.5% for those over 60 months, so it does rise with age, but quite gently. This does not contradict the AI Index's single-point spikes: the former is the long-run trend across all benchmarks, the latter is the steep early climb of a few new ones, and reading them mixed together inflates the panic. The most counterintuitive line in the same study: there is no statistically significant difference in saturation rate between public and private test sets. Hiding the questions stops verbatim leakage, but it does not stop saturation itself, because saturation has other sources: question quality, test strength, answer guessability.

Detective story three: three generations of SWE-bench falling apart. Generation one: an audit found that 32.67% of "successful" patches involved answer leakage, with the fix written out in the GitHub issue comments for the model to copy, so close to a third of the successes were copied. After cleaning out the leaked items and the weakly tested ones, SWE-Agent plus GPT-4 fell from a 12.47% resolve rate to 3.97%, leaving a third of the score once the water was squeezed out. Generation two, Verified, was the human-checked edition, and OpenAI then found at least 16.4% of tasks defective, around 59% of the hardest 138 broken, and models able to reproduce the human original patch verbatim. It was retired in 2026. Generation three, Pro, was audited within a year at around 30% broken and its recommendation withdrawn, during which top model pass rates went from 23.3% to 80.3% in eight months. Each generation's usable life is shorter than the last. The root cause is the material: GitHub issues are produced for human collaboration, the answers get written in the comments, and the tests were never designed to stop an AI. Build a benchmark out of that material and the defects are inherited. OpenAI's proposed direction is to have senior engineers design tasks from scratch specifically for AI evaluation, at the cost of being expensive and hard to scale.

Agentic evaluation has one more cheating surface that static benchmarks lack: the environment itself. An agent can use ordinary git commands to read the repository history and fish the official fix straight out of a future commit. Nobody has plugged that route, and the nearest exit is right there (this "plug one hole and the next one is beside it" mechanism has a formal name in Y05; the systematic treatment of agent evals is in Y14). The forensic dilemma, namely why after-the-fact detection cannot be relied on, is unpacked separately in Y13, and the defensive time-splitting and held-out designs are in G03.

Building questions too hard to be ground out in the short term is the other line of defence, and it demonstrated a brand new failure mode. Humanity's Last Exam collects 2500 frontier problems across disciplines, and top systems initially scored about 8.8%, though the authors themselves predicted that number would not hold for long. FrontierMath commissioned professional mathematicians to write unpublished original problems, and SOTA was initially under 2%, which looks like the ultimate anti-contamination design. Then in January 2025 the press revealed that this benchmark, sold on the promise of unseen questions, was funded by OpenAI; the mathematicians paid to write the problems were not told in advance; OpenAI held exclusive access to the great majority of the hardest problems and their solutions; and it used the benchmark to announce that o3 had reached 25% (versus 2% for the previous generation). Epoch admitted a transparency failure and said there was a verbal agreement not to train on the problems. The lesson in one line: however ingenious the technical anti-contamination design, as long as there is an asymmetry of funding and data access between the party writing the questions and the party being measured, the guarantee that "the model has not seen the questions" becomes the thing that needs auditing. This is no longer a statistical problem, it is a governance problem, and the evaluation governance line continues in Y15 and G06.

One open question: if n-gram fails, MIA fails, and semantic detection can be evaded, is "proving a score is clean" even possible in principle? The D4 report argues this pushes contamination into the same position as adaptive overfitting (Y02: repeatedly selecting on the same paper overdraws it): the worst case is undecidable, and all you have is institutional design (held-out sets, time splits, audits) rather than after-the-fact detection.

The one-line takeaway: before reading any score, ask two things, whether the questions are new and whether the questions are right; if neither can be answered, the score is a piece of PR copy.