Overfitting

Machine learning has its own name for Goodhart's law: overfitting. Statistical learning theory describes the same thing, it just never says the word Goodhart.

You have met this student. He preps for the exam by memorising past papers and their answers cold, then takes the past papers as mock exams and scores full marks every time. On the day, the paper is new. He falls apart. He never learned the pattern. He memorised that particular set of questions.

Machine learning replays the same scene. You have a batch of data (the training set, the worked examples you give the model) and you want the model to learn the pattern. Keep raising model capacity (how much it can memorise, how many bends its fitted curve can make, the degree of a polynomial for instance) and the error on the training set keeps falling, as if it were getting better and better. But test it on data it has never seen (the test set) and the error falls, then rises. Past a certain point the model stops learning the pattern and starts memorising the coincidences and noise specific to this batch of examples. A high-degree polynomial that passes through every sample point yet oscillates wildly between them is the most vivid picture of that memorisation. Perfection on the training set is bought with real generalisation (performance on new data).

The phenomenon has a clean algebraic skeleton, written down by Geman et al. in 1992 as the classic bias-variance decomposition: a model's expected error splits into three parts. Bias: the model is too simple, it never captures the full pattern, and no amount of training closes the gap. Variance: the model is too flexible, it treats the accidental texture of this particular batch as pattern, and it falls over on the next batch. Plus the noise in the data itself, which nobody can remove. As capacity rises, bias falls and variance rises, and the sum of the two falls then rises. That is where the textbook U-shaped curve comes from. Its correspondence with Goodhart is visible at a glance: training error is the proxy you can optimise directly, generalisation error is the target you actually care about. While capacity is still small, what falls is mostly bias, so training error and generalisation error move down together, in the same direction. Past a critical point, variance takes over: training error can still be pushed lower, but generalisation error turns around and rises. Proxy and target decouple right there.

Write the mapping out term by term: training set = the proxy you optimise; true distribution = the target you actually want; model capacity = optimisation pressure. "Overfitting the training set" and "inflating a benchmark score" (a benchmark being a public question bank that scores all models on the same questions) are two names for one thing.

The isomorphism goes one layer deeper. The blue branch's B08 splits Goodhart collapse into four types, two of which need nobody to act in bad faith, since pure statistics produces them. Overfitting demonstrates exactly those two. Regressional: test score = true ability + noise, so picking the highest scorer out of a pile of candidates systematically picks the one whose noise happened to run high, which means the leader's true ability is in expectation lower than its score. This is also called the optimiser's curse (B13 covers it). Extremal: push a score to the extreme of its distribution and the correlation between proxy and target breaks at that extreme, so on a benchmark pushed to 99% the final percentage point is usually no longer measuring ability. The other two types (adversarial, where a vendor games the leaderboard on purpose, and causal, where the change moves the score without improving real utility) get their scenes in Y11 and Y03.

The machine learning community grew a three-way discipline for this: the training set tunes parameters; the validation set (another slice cut out of the data) picks models and hyperparameters; the test set is looked at once at the very end, as the report card. The most overlooked role of the three is the validation set. It absorbs all the day-to-day selection: every time you tune a parameter or pick a model, you are making that decision by its score. The more you look, the more you pick up, not just the real differences between models but its own noise too. That is why it wears out first. The best configuration you picked on it is partly the optimum of exactly that noise, and that is precisely why the final report has to go back to the test set.

And "look only once" is not fastidiousness, it is priced. Dwork et al. gave the rigorous version in Science in 2015: every decision you make using the test set is an adaptive query against it, and each one consumes its statistical validity. They also gave the antidote, Thresholdout: mediate access to the test set through a differential privacy mechanism (a technique that deliberately blurs query results to cap how much information each one leaks), and you can keep generalisation guarantees across exponentially many queries. What the blurring does is plain: the reading is too fuzzy to locate the test set's noise, so the noise cannot flow into your decisions. The test set is therefore more like an account than a mirror. The balance is finite, every look is a withdrawal, and the statement never arrives in the post.

How far that account has actually been overdrawn in practice has one surprising measurement. Recht et al. in 2019 rebuilt a fresh ImageNet test set following the original protocol (ImageNet being the most important image recognition benchmark of the deep learning era) and made every model of the past decade sit the new exam. Every model lost ground: across the CIFAR-10 and ImageNet studies combined, drops of roughly 3% to 15%, about 10% on ImageNet. At first glance this looks like hard proof of a decade of collective overfitting.

But the key finding is the opposite: relative rankings were almost perfectly preserved. The drop came mainly from the mild distribution shift of a slightly harder new test set, not from a decade of the whole community iterating on one test set and overdrawing it adaptively. This detail is worth chewing on. Distribution shift docks all models in roughly the same direction, like the whole exam being swapped for a harder paper, where everybody loses a few points but the order stays put. Adaptive overfitting should instead favour the models that were selected on most often, and make them fall hardest on the new paper. That did not happen, and its absence is the evidence that the overdraft is limited.

This draws an empirical upper bound on "evals degrade with use": the ledger is real, the worst-case theoretical guarantees are real, but where the question bank is rich enough and the community advances by changing architectures rather than by pure score-chasing, the drawdown is far slower than the panic. Degradation runs at different speeds in different settings. Y02 is the fast-degrading side: when optimisation pressure concentrates on a single target, degradation turns severe. Y03 is the most literal version: the test questions go straight into the training corpus, and overfitting no longer needs any statistical mechanism, since memorisation suffices. The defensive prescription is G03's held-out and dynamic evaluation: since every look is a withdrawal, lock part of the question bank somewhere the optimisation loop can never see.

One open question: benign overfitting shows that a very large model can fit a noisy training set word for word and still generalise well, so where should the boundary of "overfitting" actually be drawn? At "fitted the noise", or at "fitted something that belongs only to this paper and evaporates on the next one"? The second way of drawing it is exactly the definition used in Y02 and Y03.

The one-line takeaway: perfection on the worked examples is purchasable, and the price is the truth on new ones; your eval is that batch of worked examples.