Held-Out & Dynamic Benchmarks

How do you keep the test ahead of the cheater? It is one move in three variants: hold a batch of items back, test only on items that appeared after release, and rotate items on a schedule. First, a cold fact: holding items back buys time, not immunity. It slows cheating down; it does not stop it.

You met adaptive overfitting in Y02: compare models repeatedly on the same set of items, pick the best each time, and you eventually overfit to the luck of that set. Y03 covered contamination: items leak into the training set, and what the score measures is memory rather than capability. Both are handled by the same idea.

There is only one idea: leave the tested party no way to know in advance, and no way to memorize, what will be tested. It lands as three moves, soft to hard:

• Keep a private set: publish which dimensions you test and by what method, but keep a batch of items unpublished, never entering any training or tuning, used once at final scoring.
• Time slicing: test only on items that appeared after the model's training cutoff. It has not seen the future, so it cannot have memorized it.
• Dynamic refresh: swap in a new batch periodically, shortening the window in which any one batch can be figured out.

There are working examples of the last two. Some benchmarks rotate in a new batch of items every month and score objectively with automated scripts, so by design there is no fixed set to memorize. Others have humans and models write items against each other, deliberately picking awkward samples the current model fails on. Patch one weakness and it hands you another. What these share is a live item pool rather than a list frozen at publication.

Two more things about the underlying principle, plus a twist. Comparing models repeatedly on the same item set and picking the best leaks information every time you pick: you are using that set's scores to steer your choice, which is a peek at the answers. Enough peeks, and the direction you steer in becomes the luck of that particular set rather than real ability. That is overfitting, and it is why a private item bank has to be strictly isolated, never entering this pick-and-compare loop. Smarter mechanisms have been designed for public leaderboards, for instance updating the published score only when your new result clearly beats the historical best, which caps the information leaked per submission. But discount the panic somewhat: a review of over a hundred data competitions found this kind of overfitting surprisingly weak, because everyone's models are highly similar, which amounts to far fewer genuinely independent attempts than the count suggests. So on the specific question of reusing one item set repeatedly, the theory is alarming and the measurements are mild. The real damage happens when you grind on a single objective and let the model see the answers.

None of the three moves is new, and all of them hit a wall on contact with reality. The history of the coding benchmark SWE-bench is a record of that. After the original came out, one team paid people to review it item by item and produced a "verified" version. A later check found that at least 16.4% of items had broken tests, and among the hardest 138 items, the ones that best separate top models, roughly 59% were broken. So a human-reviewed clean benchmark is not that clean. Worse, frontier models from several labs could reproduce, almost word for word, the fix patches humans had written back then. That is reciting, not reasoning. Then came a contamination-resistant upgrade holding part of the items back as a private bank, and a 2026 audit found roughly 30% of its items broken, with pass rates jumping from 23.3% to 80.3% in eight months. The maintainers withdrew their recommendation. Every repaired generation gets saturated, then a new one is built, and each new version has a shorter life than the last.

A private item bank has a hole nastier than contamination: an asymmetry between test setter and tested party in money and in who gets to see the items. FrontierMath, a math benchmark built from original hard problems, was meant to be the model of contamination resistance, and at first models solved under 2% of it. Media reporting later revealed that it was funded by an AI company, and that this company had access to the great majority of the hardest problems and their answers, while the six paid mathematicians who wrote items had not been told in advance. The company then used it to announce that its own model scored 25%. The issue is not whether that number is true. It is the relationship itself: the test setter is funded by the tested party and also controls who gets to see the items, so it is no longer an independent third party. The guarantee that "the model has not seen this problem" originally rested on the test setter and the tested model being mutually uninformed and kept apart. Once that independence is gone, the isolation is hollow too, and the guarantee is no longer determined by technology but by governance. This connects to G06.

A last bucket of cold water: do not treat holding items back as a silver bullet. A 2026 study dug through 60 mainstream benchmarks and found that older items saturate more readily, meaning top models can no longer be told apart, rising from 42.9% for benchmarks under two years old to 54.5% for those over five. The rise is not steep. The stinging part is elsewhere: public item banks and private held-back banks showed no statistically significant difference in saturation rate. Hiding items does not prevent saturation. What hiding does is push the moment of being figured out further out, buying you a window to swap and rotate. It buys time, not immunity. The real defence is running all three moves as a pipeline that never stops turning, not hoping one batch of hidden items stays clean forever.

When designing an eval, work through this list:

• Keep a private set for final adjudication only, never entering the training or tuning loop.
• Use time slicing where you can, favouring real tasks generated after the model's training cutoff.
• Rotate items on a schedule, giving each set a service life and swapping it when that expires.
• Run contamination audits proactively, and retire items whose answers can be recited word for word or whose scores drop when you reorder the options.

One open question: what share of frontier models' high scores is contamination? One study gives a counterintuitive answer: a single leaked item entering pretraining should, by rights, be devastating. But training on vast amounts of new data afterward dilutes it, even erases it. So large models barely lose points after massive training; small models lose a lot. But we still have no cross-benchmark, cross-vendor accounting that says how much of a high score is contamination, how much is scaffolding, and how much is real capability.

The one-line takeaway: hidden items, time slicing and scheduled rotation are a pipeline that has to keep turning; it buys time, not immunity.