Adaptive Overfitting

Looking is a form of training. Adaptive overfitting says: you never trained on the test set, you only looked at it repeatedly and used it to select models. But looking at the same set of questions over and over to make choices is itself a filter: what you keep is always whichever version happened to perform best on this particular batch of questions. Over time, that is how the paper's idiosyncrasies get poured into your model.

You have probably lived this. You build an eval to measure how good your agent is, use it over and over to filter, tune and pick the best version, and the eval score climbs beautifully. In real use, nothing has changed. The model only learned to fit your eval snugly. This is not a personal failure: any process that uses the same eval as both the selector and the report card is structurally re-enacting this move, and the more times you select, the more distorted the report card gets.

The move has a formal name in statistics: adaptive data analysis. The distinction takes one sentence. Non-adaptive: every question you will ask of the data is listed before you see the data, and all the guarantees of classical statistics rest on that premise. Adaptive: the next question depends on the last answer. Every time you decide what to change next based on the last round's score, you are doing the latter, and the classical guarantees fail wholesale.

Two pieces of work from 2015 turned this into accounting. Blum and Hardt designed the Ladder algorithm for Kaggle-style leaderboards (Kaggle being the data science competition platform where entrants submit models repeatedly and watch their rank), and named the pathology bluntly: participants are allowed to evaluate their submissions on the leaderboard repeatedly, and so may start to overfit. Their fix updates the public score only when a new submission is significantly better than the historical best: if the score does not update, the participant has effectively received no feedback, and this submission leaked nothing usable; only a submission that truly, significantly beats the historical best earns an update, and the number of such wins is bounded. That is how the information leaked per submission gets capped. Dwork et al., in Science, gave the more general pricing: mediate access to the holdout through a differential privacy mechanism (covered in Y01: deliberately blur the reading, cap the per-query leak), and you can support exponentially many adaptive queries without invalidating it. Together the two established an accounting view: the test set is a budget, a query is a spend, and the cost of looking has to be booked.

The simulator above is worth a few minutes. Run the defaults to the end and the leaderboard score (the glowing line) rockets, while "full-dimensional true ability" (the red line, what users actually experience) barely moves. The glowing gap in between is the illusion, and it has two parts:

• Question overfitting: the model has memorised the tricks, coincidences and phrasings of this specific set of questions. Swap in a new set and this part vanishes instantly. Switch "question bank policy" to "new questions each generation" and you will see it zero out on the spot. The real-world counterpart is benchmark contamination: GSM8K (the most-used grade-school word problem benchmark) had its questions eaten into training sets, and when it was replaced with GSM1k, freshly written in the same style, a batch of models dropped on the spot (details in Y03).
• Dimensional narrowing: the model really did get stronger, but only on the small slice of dimensions the eval covers (10% in the simulation's defaults). It is legitimate but partial. You are training where the streetlight shines.

Now set "capacity constraint" to zero-sum, the setting with the heaviest consequences. In reality, model capacity, data mix and RL compute are all finite, and what you spend on the measured dimensions has to be taken from somewhere else. You will watch ability on the unmeasured dimensions genuinely go negative: the model is not merely failing to improve elsewhere, it is actively regressing, reshaped by the eval's contours. This is the cleanest picture of "the more evals, the more the AI degrades into the eval".

To see the gap between theory and experience, look first at the shape of the evidence. Every Kaggle competition has a public and a private leaderboard: the public one gives day-to-day readings and is queried constantly, the private one is revealed only at the close and is looked at once. That is a natural control group. Roelofs et al. measured the distribution of the gap between the two boards across a hundred-plus competitions, and the gap turned out to be explained mainly by random fluctuation, not by systematic adaptive overdraft. What that result says: the whole community ground the public board for months and left no trace of a collective faceplant on the private one.

Why did the theoretical worst case not materialise? Ladder and the reusable holdout defend against an idealised adversary that extracts maximum information from every query, and real leaderboard grinders are far less efficient than that: everyone's models resemble one another, and a thousand submissions may contain only a few dozen effectively independent probes. There is corroboration too: Miller et al. re-examined question-answering benchmarks in 2020 and likewise found no evidence of adaptive overfitting, with the drops coming almost entirely from the natural drift of slightly harder new questions. On that basis the Roelofs group wrote that the holdout method is quite robust across data domains, loss functions, model classes and analysts. This line of work now has its own programme: Hardt in 2026 made the benchmark itself the object of study, calling it the emerging science of machine learning benchmarks, and Y15 at the end of the yellow branch comes back to it.

So the map splits in two. On the classic supervised learning half, the question bank is rich enough, the optimisation objectives are diffuse, and the community advances by changing architectures rather than pure score-chasing, so degradation is diluted to nearly unmeasurable. The other half is where optimisation pressure concentrates on a single memorisable target: RLHF grinding on the human preference proxy, which produces sycophancy and padding (Y07, Y08); agentic coding grinding on a fixed question bank, where frontier models have been audited reproducing, word for word, the human-written fix patches inside SWE-bench (Y03). Between the halves there is also a man-made amplifier: a few vendors privately test many variants before releasing on LMArena (the model leaderboard voted on by users in blind comparisons) and publish only the best one. All those variants come from that same single vendor, unlike a public competition where many independent entrants dilute the pool. That concentrates the community's diffuse adaptive queries into a single pair of hands (Y11). On the concentrated half, "degrading into the eval" is proven. To judge whether the eval in your hands is dangerous, look at two variables first: how concentrated the optimisation pressure is, and how memorisable the target is.

If you can notice that "nothing changed before and after the eval", the noticing is worth more than the eval. Level one: all eval lights green counts as progress. Level two: you see the failure of the measurement itself. Level three: you build an evaluation system that knows from the start it will be gamed and is designed for it.

One open question: the Kaggle verdict comes from supervised learning competitions before 2019, when "model selection" meant submitting a file of predictions. Today you tune prompts, swap scaffolds and pick checkpoints against the same eval, and the shape of the adaptive query has changed entirely. Whether the classic verdict extrapolates has no re-examination of comparable scale. The other half of the question is more twisted: model similarity is both protecting the holdout (fewer effective queries) and manufacturing ecosystem homogeneity (everyone distilling from the same few strong models). The same fact has opposite signs on two ledgers. How should it be booked?

The one-line takeaway: the test set is an account, not a mirror, and every look is a withdrawal; the classic settings just debit slowly, while settings that grind on a single target debit fast.