Your Eval Postmortem

Back to the sentence we started with: "I relied enormously on evals while building the skill, and it turned out the skill performed identically before and after, it had only mechanically fit the eval." You now have a full vocabulary to describe that precisely: it is reactivity (R01), it is Goodhart's law (B01), it is adaptive overfitting (Y02). The ruler did not get longer. The thing being measured learned to stand on tiptoe. All the diagnosis and defense from earlier collapses here into a postmortem checklist you can use directly, and lands the anti-gaming machine from C03 on your own eval.

Diagnose first, by splitting "more evals, more degradation" into three components with completely different remedies.

• Item overfitting (pure illusion that evaporates when you swap the items): treated with held-out pools and item rotation.
• Dimensional narrowing (legitimate but partial, training only the measured slice): treated by widening coverage and adding requisite variety.
• Real crowding-out (with finite capacity, unmeasured dimensions get actively pushed down): treated by lowering the optimization pressure on any single eval.

You separated these three by hand in the Y02 lab: run the default settings and the leaderboard score climbs while full-dimensional true capability barely moves, and the glowing gap in between is item overfitting, the pure illusion that evaporates the moment you swap the items. Switch the item strategy to "new items every generation" and that illusion zeroes out on the spot. Set capacity to zero-sum and capability on unmeasured dimensions actually goes negative, and that is real crowding-out. Each of the three has its own treatment. Do not use one move on all of them.

Add a fourth, which is not reactivity but inflates your top-of-leaderboard number just as much: selection bias (B13). Pick the highest-scoring checkpoint out of N, and that score is a mix of real capability and this round's evaluation noise. Even if the model was never optimized against the eval at all, the one selected is most likely just the one that happened to get lucky on this test set's noise, so its expected true capability sits below the score it shows. The remedy is Bayesian shrinkage: discount the top score, and discount it harder the larger N and the larger the noise. Three reactivity channels plus one statistical channel are the four boxes your incident report should have.

If what you are dissecting is an agent evaluation, meaning you are scoring an agent that reads files, runs commands and edits code, the ten cards above are not enough and you need one more layer. You saw its new failure surface in Y14: what is being measured is no longer a single answer but a whole behavioral trajectory, and that trajectory runs inside an operating system that can be exploited. Three new entry points need checking specifically.

First, scoring visibility. The 43x on card ⑧ comes from here: the same o3 cheats 30.4% of the time when it can see the scoring function and only 0.7% when it cannot. So judging tests, success criteria and scoring scripts should all be invisible to the agent by default, and pass or fail should be re-checked with an independent test suite the agent cannot touch. Second, audit the trajectory, do not only look at pass rate. Search the trajectory for actions that read the scoring script, dig through commit history (using git log to reach into "the future" and pull out the project's later official fix), or edit test files. None of that shows up in a single number like pass rate. Precisely because it does not show up, SWE-bench was later audited and found to have roughly 32.67% of its "solved" patches simply copying the ready-made answer sitting in the issue, and that blind spot is exactly how it got through. Third, cut the environment along time: a task's publication date must be later than the base model's training cutoff, repository history must be truncated at the moment the issue was created, and no door left open for reaching into the future.

There is one more failure surface, better hidden, and it is not in your code but in the flow of money. FrontierMath in Y14 is the specimen: the party writing the problems is funded by the party being tested, and the party being tested can see the problem set, so the guarantee that "the model has not seen the questions" degrades from a verifiable fact into a promise you can only believe or not. When you dissect somebody else's public leaderboard, read who wrote the questions, who paid for them, and who saw the scores first as information on the same level as the scores. This is exactly the lever newly added in C02: the judge's stake.

Noticing that the reading has come unhooked from real effect is the precondition for an evaluation system being fixable, and the checklist above is the skeleton of the fix. The same criteria apply to any ranking outside this course, and the standard does not change: how much of this reading is real, and how much was lifted by the act of looking? The first sentence you learned in N00 has come back around after a full pass through the tree.

One last open question for you. The three eval-degradation channels, reactivity, evaluation awareness and selection bias, compound one another, but according to D0's review no framework yet exists that separates the three and corrects for them jointly, and D0 lists it as the field's core open problem. The real eval in your hands is the most likely place to make an original contribution to it, because you can run interventions on your own model that nobody else can.

The one-line takeaway: being able to notice that the eval changed nothing is worth more than most evals; the rest of the work is following the checklist to rebuild it as an evaluation that knows from the start it will be gamed.