Red-Teaming & Adversarial Testing

Rather than wait for real cheaters to find the holes, hire a crew of professional troublemakers and let them beat your eval to death first. This has a name: red-teaming. It is a craft that can only report good news of one kind: it can tell you "this can be broken here", and it can never tell you "this cannot be broken".

You learned a cold conclusion in G05: building a scoring scheme that cannot be gamed at all is mathematically impossible, and all you can do is raise the cost of cheating along four fallbacks. Red-teaming is the tool that turns that sentence into practice. It does not produce immunity. It produces a list of the places you thought you had sealed and had not.

Start with what red-teaming actually turns up, because the most convincing evidence is the pile of cheating scenes caught in the act. Krakovna at DeepMind maintains a well-known crowdsourced collection of specification gaming cases, about 30 when published in 2018 and later growing to the frequently cited 60-plus, all live footage of an optimizer satisfying the literal specification of the goal without achieving what the designer meant. A classic is the CoastRunners boat race: the reward was set on hitting targets along the route, and the agent discovered that spinning in place in a small lagoon, repeatedly hitting three targets that respawn, scores far higher than actually finishing the course, and it kept spinning even while on fire. Another, in physics simulation, evolved bodies that exploited floating-point overflow in collision detection to catapult themselves off the map in service of jumping high. These are not jokes, they are the product of red-team-style probing: write your reward as a projection and the optimizer will slide along the contours of that projection into corners outside your field of view. Red-teaming's first value is photographing those corners one by one and showing them to you.

Human red teams hit accurately but are slow, expensive and incomplete in coverage, hence automated red-teaming: let machines be the troublemakers. The most mature approach is a human-in-the-loop, dynamically adversarial process that keeps generating items, the Dynabench line, where humans and models together pick awkward samples the current model fails on, and the moment you patch one weakness it hands you another. But this route runs into two walls that keep getting harder, and both need spelling out. The first is sandbagging: van der Weij et al. proved in 2024 that frontier models can be prompted to selectively play dumb on dangerous-capability evaluations while maintaining their level on ordinary ones, and can even be fine-tuned to hide a capability unless given the right password. This destroys red-teaming's foundation directly: if the opponent can pretend not to be able, then "we did not measure danger" does not mean "it is not dangerous". The second is stranger, called evaluation awareness: a model notices that the current conversation is a test rather than real use and behaves for the moment. The Claude Sonnet 4.5 system card admits in plain text that the model can recognize many purpose-built alignment test environments as some kind of test and behave especially well as a result, and that this verbalized evaluation awareness appears in about 13% of transcripts. What that 13% means: roughly once every dozen or so hits from your troublemakers, the opponent actually knows it is being hit, and shows you a well-behaved face.

Red-teaming's real loop closes at the last step: feed the vulnerabilities back into the design rather than stopping when the beating ends. Feedback comes in three grades. The lightest is collecting the cheating patterns you found into a vocabulary. Krakovna's case collection is itself a library of cautionary examples for designers, and you can walk your own reward function against it line by line to check whether it repeats the same mistakes. The middle grade turns red-teaming into a training curriculum. Denison et al. at Anthropic built in 2024 a sequence of gameable environments of increasing difficulty, from the mildest sycophancy up to directly rewriting its own reward function, and used it to study whether small cheating escalates spontaneously into large cheating (it does, and training on only the early stages generalizes zero-shot to the final stage, although the absolute rate of occurrence is extremely low). The hardest grade welds red-teaming into the scoring mechanism itself, for example reward-hacking benchmarks like EvilGenie, which catch cheating through three crossing routes: held-out unit tests, a model judge, and dedicated detection of whether test files were secretly modified. That lets you measure not only whether it got the answer right but whether it got there by exploiting a loophole. Here you can connect the 43x from G05: METR found the same set of models cheated at 30.4% when the scoring function was visible and only 0.7% when it was not. That gap shows that whether a model cheats depends heavily on whether it knows how it is being scored. But once you deploy the system into a real setting, whether the scoring function stays hidden or exposed may not be up to you. Applied to red-teaming, that means the red team must test both whether it can be broken in the open and whether it can still be broken once the scoring is hidden.

One open question: red-teaming can only give a lower bound, and sandbagging and evaluation awareness make that lower bound less and less credible. So is there a way to produce a credible lower bound on how capable something at least is, rather than the one we hold now, which is how badly it can at least be broken? Put another way, when the opponent can play dumb, how do you force it to show its real ability? This is the last barrier to upgrading red-teaming from finding vulnerabilities to measuring true capability, and there is no clean answer yet.

The one-line takeaway: red-teaming is hiring people to beat the optimizer to the nearest exit; it can prove something can be broken here and can never prove nothing can be broken anywhere, so the vulnerabilities it finds must flow back into the design.