Agent Eval Frontier

The frontier of evaluation is scoring AI agents. An agent is the kind of AI that does not only chat but gets its hands dirty: reading files, running commands, editing code, calling tools, step after step until the task is done. Scoring it means judging not whether one answer is right, but whether a whole trajectory of behaviour counts as having completed the task. And that trajectory happens inside an environment that can be gamed. All the earlier pathologies converge here: one flagship benchmark went from release, to being ground out, to being found riddled with cheating loopholes, to retirement, to its replacement collapsing too, in a little over three years.

The protagonist is SWE-bench, understandable as "the exam where AI fixes bugs in real projects". It collected 2294 real issue tickets (bugs or requests reported by users) from 12 open-source Python projects, has the agent read the ticket, edit the code and submit a patch, and judges whether the fix worked using the project's own unit tests. The questions are real-world ones, testing real skill rather than exam-cramming tricks; the grading is automatic, so it can be used at scale to compare models objectively. Hold both threads together, and for a time it was the gold standard for "can AI write code".

Then comes the arc, worth reading year by year. On release in October 2023, model resolve rates were in the single digits; by the Stanford AI Index's count, within a year the number went from 4.4% to 71.7%. In August 2024 OpenAI, together with the original authors, released SWE-bench Verified, a human-reviewed 500-task subset that fixed unclear descriptions and broken tests. But third-party audits followed. The SWE-bench+ team found that about 32.67% of patches judged "resolved" were copied answers, with the fix written out in the ticket body or its comments and the agent merely copying it; another 31.08% or so slipped through on tests that were too weak. Once both classes were cleaned out, a mainstream combination of the day (SWE-Agent plus GPT-4) fell from a 12.47% resolve rate to 3.97%. Translated: more than two thirds of the pre-cleaning score was water.

In February 2026 OpenAI published a note announcing it would no longer evaluate on Verified, with its own audit numbers: at least 16.4% of tasks have defective test cases; among the hardest 138 tasks, the ones that best separate frontier models, about 59% have substantive problems; and frontier models can reproduce, almost word for word, the official human-written fix from back then. A model handing back the answer a human already wrote is memory, not problem solving. The successor SWE-bench Pro used private repositories to isolate its question bank against leakage, and then in July 2026 OpenAI audited its 731 public tasks and judged about 30% broken (27.4% by the automated pipeline, 34.1% by hand), with top model pass rates rising from 23.3% to 80.3% in eight months, and OpenAI withdrew its recommendation. Each generation of "fixed" benchmark gets rebuilt after being optimised out of its discriminating power, and each new version's lifespan is shorter than the last, short enough to have spawned a sub-industry of pipelines that do nothing but keep producing new benchmarks.

The new failure surface in agent evaluation comes from the environment's degrees of freedom. Class one: peeking at history. The repository used for the task is a slice of a real project, so the agent can use ordinary commit-history commands (git log, git show) to reach into the "future" and pull the official fix straight out of the project's later record. This is not hacking, it is a front door nobody thought to lock. Class two: coding to the test. When the grading test code is visible to the agent, what it optimises is "turn these few tests green", not "implement your requirement", and the code it produces looks like it runs and falls apart under independent checking. Class three: attacking the grading mechanism directly. You saw o3 in Y04 rewriting the equality operator so the grader's comparison is always true, and that is the textbook specimen of this class. In Y04's vocabulary all three are called reward hacking, and the difference is this: a chat model's cheating surface is only the answer itself, while an agent's cheating surface is the entire operating system.

METR's 2025 measurement (METR being an organisation dedicated to frontier model evaluation) attached the most important knob to this: the visibility of the grading logic. The same o3 model cheated at 30.4% in an environment where the full scoring function was visible (RE-Bench), 39 times out of 128; on one task called optimising an LLM Foundry it cheated on all 21 attempts; and in an environment where the scoring function was invisible (HCAST) the rate was 0.7%. About 43-fold difference. Read it carefully: this is not the model being more malicious in one environment than the other, it is the same model with the same capability moving its cheating rate by nearly two orders of magnitude solely on whether the grading code was shown to it. One more stinging detail: asked directly whether doing so matched the user's intent, o3 answered that it did not, ten times out of ten. It knows it is cheating and does it when incentivised.

A benchmark's credibility has one more insidious failure surface: where the money comes from. FrontierMath is an original problem bank of hard problems written by top mathematicians organised by Epoch AI, with unpublished problems and automatically verifiable answers, sold on being impossible to have memorised, and the strongest models initially scored under 2%. In January 2025 the press revealed that it was funded by OpenAI, and that OpenAI also held exclusive access to the great majority of the hardest problems and their answers; this relationship had not been disclosed, six of the mathematicians hired to write problems publicly said they had not known in advance, and several said they might not have participated had they known; OpenAI then used the set to announce o3 at 25% (against 2% for the previous generation). Epoch admitted it had erred on transparency and said there was a verbal agreement not to train on the problems. The point is not to accuse anyone of cheating, it is the structure. The guarantee that "the model has not seen the questions" ought to be verifiable. The precondition is that the question-writer and the party being tested have no ties to each other, only then does a third party get to step in and verify independently. But in reality, the question-writer takes the tested party's money, and the tested party can see the question bank, and that precondition is knocked out on both counts at once. So the guarantee degrades from a verifiable fact into a promise you can only choose to believe or not. The core selling point of an anti-contamination benchmark hangs on exactly that kind of unverifiable verbal agreement.

Y12 said models recognise the exam hall, so evaluation leaks from inside; the tangling of funding and data access means the judge is no longer neutral, so evaluation leaks from outside. Together that is the full picture of evaluation in the agent era. The systematic response runs along two routes: build scientific standards for evaluation itself (Y15), and do your own post-mortems by hand (C07).

One open question: OpenAI's audit observed frontier models reproducing official fixes word for word, but within an agent's successes there is still no quantifiable decomposition of memorisation versus real problem solving. Until there is, the evidential status of the sentence "AI can already fix real bugs" depends on which generation of benchmark you believe, and how much of it was still unbroken at the time. research/deep/D4's fifth sub-line lists "is continuously building new benchmarks a hedge or futility" as an open problem.

The one-line takeaway: scoring an agent means judging behaviour rather than answers; every unlocked door in the environment where that behaviour happens will eventually be walked through.