Contamination Forensics

Y03 said exam questions get eaten into training sets: once questions leak into training data, the score measures memory rather than ability. The next question is forensic: given a model and a score, can you tell whether it memorised the questions, and how much memorisation inflated the score? The answer is uncomfortable. The mainstream detection methods essentially fail: that is the first reason. The effect of a leak also drifts as training proceeds: that is the second reason. Stack the two together, and "proving a score is clean" may be impossible in principle. This determines the posture in which you should believe any score at all.

Lay out the forensic toolkit first, three layers of it. Layer one checks verbatim overlap: cut the test questions into small fragments (the term is n-gram, meaning a run of consecutive words) and search the training corpus for exact matches; existing infrastructure can already run such retrieval over trillion-token corpora. Layer two is used when the training corpus is unavailable, and is called membership inference: infer from how "familiar" the model is with a question (answering it with unusual fluency, say) whether it saw that question during training. Layer three is a behavioural control: write a whole new set of questions in the style of the originals, have the model sit them again, and the size of the drop is behavioural evidence of the memorisation component. Of the three, the first two have been shown to fail essentially. Only the third stands up. But that third layer is expensive and slow, doable only one benchmark at a time.

Layer one's failure is adversarial. Yang et al. ran a simple experiment in 2023: rewrite the test questions (reword them, or translate them into another language) before mixing them into the training set. Standard screening based on verbatim overlap and vector similarity let all of it through, and the scores were still substantially inflated. Their conclusion: only a large model doing semantic-level comparison catches rewrite-style leakage. Dekoninck et al. put it more coldly in 2024: the cost of evading every existing contamination detector is so low that the paper's title carries the sarcasm, it is (too) easy. Whatever detector the defence uses, the offence can bypass it cheaply.

Layer two's failure is statistical. Duan et al. systematically tested membership inference on large language models in 2024 and concluded it is barely better than a coin flip: training data is too large, and a given question is seen only a handful of times. The trace those few exposures leave is diluted in the model's weights below detectability. Meeus et al.'s 2025 systematic review titled this line of work rushing nowhere, and exposed a more embarrassing fact: many studies claiming to have "detected contamination" were measuring something else. The "unseen questions" they used as controls were usually newer than the "seen questions", and models are inherently less familiar with content from a later period. This time gap gets mistaken for a memorisation signal. In other words, a large batch of public accusations that "model X contaminated benchmark Y" do not hold up methodologically.

Even with a leak confirmed, "how many points did it add" has no fixed answer. Schaeffer et al. in 2026 quantified the effect of leakage directly with a causal framework, and their conclusion is path-dependent, summarisable as a paradox. On one hand leakage is fatal: even a single copy of a test set mixed into the pretraining corpus (the first stage where a model learns internet-scale text) is enough to push the model's performance on that paper beyond what genuine ability could reach. On the other hand leakage is fragile: continue training on plenty of fresh data after the leak and the effect is diluted or even erased; the later fine-tuning stage may amplify it or dilute it further, depending on how deep the earlier leakage went. The same leak, placed at different points on the training timeline, ends completely differently.

This paradox neatly explains the famous GSM1k observation from Y03: why, after new questions were written, frontier lab models barely dropped while certain small model families dropped 8% to 13%. Frontier models had an enormous amount of subsequent training pressed on top of any leakage, so early contamination was washed out; small models had their contamination closer to the final version and the traces were still warm. So the linear intuition of "the drop equals the water added" is wrong: contamination damage is determined not by leaked-or-not alone, but by the whole training history.

This also explains why a rebuilt control is currently the only forensic method that stands up, and what exactly it measures. GSM1k wrote 1000 entirely new problems in the style, difficulty and solution length of the grade-school maths benchmark GSM8K, and each model's drop correlated with its probability of reciting the original problems verbatim, so dose and response line up. That is the gold standard of behavioural forensics. Apple's GSM-Symbolic sharpened the knife further: swap only the numbers in the problems and most models' scores slide noticeably; insert one sentence of seemingly relevant but actually irrelevant filler into the stem and the most advanced models drop by up to 65%. One extra irrelevant sentence should not touch the answer at all. Real reasoning ought to ignore it completely. But the model's score collapses anyway, and that shows what it relies on is not reasoning but the problem template. This widens the forensic object from "memorised this question" to "memorised this kind of question": a model may not have seen a single character of this problem while having seen ten thousand isomorphic ones, and verbatim detection is entirely blind to that. The concept of soft contamination proposed in 2026 pushes this to the end: even with zero verbatim leakage, if the training data distribution is too close to the exam questions, what the benchmark measures is only shallow generalisation. At this point the contamination problem merges with Y01's definition of overfitting: what really needs investigating is not "was this string seen" but "how much of the score evaporates when the questions change".

Stack the three failures together, and the position of "proving a score clean" is close to a logical dead end: verbatim detection can be bypassed by rewriting, membership inference has no statistical power, semantic detection can be evaded further still, and behavioural controls can only be done by hand, one benchmark at a time. The reasonable conclusion is not to keep hunting for a stronger detector, but to admit that the structure of proof itself is wrong: cleanliness cannot be proven, and all you can do is make contamination hard, expensive and unprofitable through institutional design. There are three routes out. One, a held-out private set: lock part of the question bank away so it never enters anyone's optimisation loop. Two, a time split: evaluate only on questions newer than the model's training cutoff, the way the coding benchmark LiveCodeBench filters by publication date and LiveBench replaces about a sixth of its questions every month, letting the timeline serve as a whitelist, since new questions are physically impossible to have memorised. Three, write contamination auditing into the benchmark's publication standards. But book this against the evidence too: a 2026 systematic study covering 60 benchmarks found no statistically significant difference between public and private test sets in the fraction that had been ground down to losing discriminating power. Private-set questions are hidden away and should be more resistant to this. But they lose discriminating power just like the public sets do. That shows hiding questions stops verbatim leakage, but not a whole ecosystem adapting itself toward the benchmark. The full version of the institutional defence line is unpacked in G03.

One open question: within the inflation of frontier model scores, what are the shares of contamination, of evaluation scaffolding (prompt templates, answer extraction and other peripheral settings) and of real ability? Schaeffer gives the mechanistic answer that it depends on the training lifecycle, but a cross-benchmark, cross-vendor decomposition of shares has no accepted result to date. Until one appears, "how much of this score is real" can be answered only up to an interval for any single model. research/deep/D4's second sub-line lists it as the number one open problem.

The one-line takeaway: you cannot determine whether a score is clean; what you can do is design the evaluation so it cannot get dirty, or assume it already is before reading it.