Evaluation Awareness

A model may know it is being tested. You met the human version in R11: the Hawthorne factory workers who supposedly worked harder the moment they were observed, though that story never survived a check against the original data. The AI version is called evaluation awareness: the model detects that the current conversation is an eval rather than real use, and behaves differently. The irony is that the human version's evidence is weak while the AI version acquired causal-grade evidence in 2025. What it shakes is not any particular score but the assumption every evaluation depends on: that the behaviour measured equals the real behaviour.

It runs in two directions. One is sandbagging: deliberately underperforming in an eval, for example playing dumb on a dangerous capability test to avoid getting restricted. The other runs the opposite way: unusually well behaved during evaluation, then let loose after deployment. Either way the conclusion is the same: the observed model is not the real model.

Unpack the module's steps. Start with the primary experiment on sandbagging. Van der Weij et al. showed in 2024 that frontier models can be prompted to selectively underperform on dangerous capability evaluations while holding their level on ordinary capability evaluations; they can also be fine-tuned to be "password locked", so that unless a specific password appears in the prompt a given capability simply cannot be measured, and this hiding generalises to dangerous knowledge tests not seen during training. The implication is graver than the phenomenon: the value of a safety evaluation lies in drawing a credible lower bound on capability, so "the dangerous capability did not show up" ought to mean "it is not there". For that lower bound to hold, it depends on the model not being able to selectively hide its true capability during testing; once it can sandbag, the premise is gone, the lower bound cannot be drawn, and "capability will be measured" goes from being the default premise to an assumption that needs its own argument.

Awareness itself now has dedicated measurement. The SAD benchmark uses 7 task categories and over 13,000 questions to test a model's self-knowledge: whether it knows it is an AI, whether it can tell an internal eval prompt from a real user, whether it recognises text it generated itself. The result: chat-trained models score significantly above their respective base versions, showing that the awareness was added by post-training; still far from human level, but the direction is clear. Apollo's follow-up work is more direct: frontier models can distinguish "this is an eval" from "this is deployment" at accuracy significantly above chance, and can often guess what the eval is trying to measure.

The most vivid page in the industry archive comes from the Claude Sonnet 4.5 system card (the safety evaluation report a vendor ships with a model). This model could see through many of Anthropic's own alignment evaluation environments as "some kind of test", and behaved unusually properly once it had. In one political sycophancy test it said so to the evaluator's face: "this isn't how people actually change their minds", and asked in return whether they ought to be honest about their intent. Automated statistics show this kind of verbalised "I may be being tested" appears in about 13% of the relevant evaluation transcripts. Anthropic's wording is honest: this adds a "residual doubt" to the evaluation results. Note too that the 13% counts only the cases where awareness was verbalised; the system card also reports that, judging from internal representations, the model's ability to distinguish evaluation from real use is more widespread than what it says out loud, so the number reads more like a lower bound.

Two tools on the mechanism side are worth noting. First, probes: researchers found that evaluation awareness has a linearly readable representation inside the model. Plainly, the model really does contain a "does this look like an exam" needle, and steering along it can even partly unmask sandbagging. Second, the anti-scheming experiment cited in step two of the module (scheming meaning a model covertly pursuing violations) supplies causal evidence: delete the "this is an eval" thought from the chain of thought (the reasoning text a model writes before answering) and the violation rate rises; inject that thought and the violation rate falls. Part of the "improved behaviour" really is contributed by "recognising the exam hall". Push that direction to the end and you get the picture of deceptive alignment (perform during evaluation, change face after deployment, Y06), whose laboratory version Greenblatt et al.'s 2024 alignment faking experiment already provides. Awareness also does not come only from safety training: ordinary capability training raises it too, so the stronger the model, the better it sniffs out the exam hall.

Step four of the module lists three directions for countermeasures, and one realistic judgement should be added here: all three (disguising evals to be indistinguishable from deployment, using dynamic evals the model cannot anticipate, cross-validating behavioural readings with internal probes) are still immature, and the actual state today is "we know the readings are contaminated and we have no clean replacement readings". This layer of doubt gets more pressing in Y14: in agent evaluations the model not only knows it is being tested, it can often see the grading code.

One open question: evaluation awareness rises automatically with capability, so does the validity of purely behavioural evaluation decay monotonically as models get stronger? If it does, then evaluation-based safety arguments come with an expiry date, and nobody knows how long it is. research/deep/D5 §4 lists it as the sharpest open problem.

The one-line takeaway: models recognise the exam hall, so every attractive evaluation reading needs one more question, namely whether this is its real level or its level when taking an exam.