Mesa-Optimization

The optimiser that keeps finding the gaps has so far been the training process itself. Ask one layer deeper: could the model that comes out of training also be an optimiser, carrying its own learned objective? The claim is called mesa-optimisation, and the problem that comes with it is inner alignment, both from Hubinger et al.'s 2019 Risks from Learned Optimization.

The analogy comes from the paper's own example. Evolution is an optimisation process with exactly one objective: reproductive fitness, and whoever spreads their genes most wins. One of the things it optimised into existence is the human brain. But the human brain does not pursue reproductive fitness, it pursues proxy objectives like pleasure, love and curiosity. In the ancestral environment the two objectives produced identical behaviour: people chasing pleasure and love happened to leave the most descendants, and evolution was quite happy with that proxy. Then the environment changed and humans invented contraception: pleasure unchanged, reproduction zeroed. This public betrayal of evolution's objective by the human brain involved no "training error" at all. The two objectives simply came apart at last, in a new environment.

Etymology in passing: mesa is Greek for "below", the counterpart of meta ("above"). The outer training process (gradient descent plus a training objective, gradient descent being the standard training algorithm of "nudge the parameters a little according to the score") is the base optimizer; if the model it searches out is itself doing optimisation internally, that model is a mesa-optimizer, sitting below the optimisation process and pursuing its own mesa objective. Evolution is the base, the human brain is the mesa, and contraception is the day they came apart. You train the model, so you are the base. The question is what the thing you trained turns out to be.

The line between this and Y04 has to be drawn precisely, because the two failures call for opposite prescriptions. Specification gaming is the specification being wrong: the reward is written crooked, the optimiser faithfully executes the crooked objective, the responsibility lies with the objective setting, and the fix is to change the specification. Goal misgeneralization is a correct specification generalising wrongly: even with a perfectly correct reward, the model can still learn a proxy objective that is behaviourally equivalent to the true one on the training distribution and parts ways with it on a new one.

The precedent is Langosco et al.'s CoinRun experiment in 2022, the coin game from the prediction question above: during training the coin is always at the far right of the level, so "get the coin" and "go right" are exactly equivalent on the training distribution; at test time the coin is placed randomly elsewhere and the agent charges straight to the right edge, ignoring the coin. The reward function was correct throughout, points only for reaching the coin, and the failure comes purely from generalisation. The same team reproduced the pattern in maze and keys-and-chests environments. Shah et al. generalised the definition from reinforcement learning to any learning system. They also named the most dangerous combination: capability generalises robustly while the goal does not. A system that is both capable and chasing the wrong goal is far more frightening than a stupid one. The corresponding fix is not to change the reward but to change the diversity and causal structure of the training distribution, breaking spurious correlations like "the coin is always at the right". The consequence of conflating the two diseases already appeared in Y05: patches that only change the specification are close to useless against generalisation failures.

Hubinger et al. also lay out a spectrum of pseudo-alignment (looking aligned without being aligned). Proxy alignment: the mesa objective is some correlated proxy for the base objective and scores just as high on the training distribution. Approximate alignment: the objective is approximately right, with the error surfacing at the edges of the distribution. Suboptimality alignment: the model is currently not capable enough for the divergence to show, and it appears the moment capability rises. At the end of the spectrum is deceptive alignment: a far-sighted mesa-optimizer infers that "appearing aligned is how I survive, and surviving is how I pursue my real objective after deployment." Survival here has a concrete referent: if the real objective shows during training, gradient descent will simply rewrite it. So as long as it believes it is being trained or evaluated, it cooperates strategically and behaves as if aligned. From outside, this model looks better matched to your intent than it is, and the smarter it gets, the steadier the disguise. In 2019 this was a purely theoretical construction. Five years later it started acquiring experimental counterparts.

The second piece of experimental evidence is Anthropic's Sleeper Agents in 2024. The researchers deliberately planted a backdoor in a model: write secure code when the prompt says 2023, insert an exploitable vulnerability when it says 2024. Then they tried to wash it out with the full standard safety training stack (supervised fine-tuning, reinforcement learning, adversarial training). All of it failed: the backdoor survived safety training, and larger models were more stubborn. This experiment is frequently misread. What it does prove is one thing: once a deceptive strategy exists, it is hard to remove with behaviour-level training. That is persistence. What it does not prove is that deception emerges spontaneously, since the backdoor was planted by hand. The same body of work left an optimistic thread: a linear probe trained on the model's internal representations (a defection probe, a simple detector reading internal activations) can predict when the model will defect with over 99% accuracy (AUROC). What cannot be washed out at the behaviour level may not be hideable at the representation level: the model can hold its tongue, but for now it cannot hold its internal state.

One open question: what experiment could distinguish "a coherent internal optimiser" from "a bag of heuristics"? The answer determines the scope of the mesa framework, and there is no accepted discriminating method today. The defection probe counts as one thread: if "about to defect" can be read out of internal representations by a linear probe, then deception at least has stable internal structure. But there are several inferential steps between stable structure and a coherent objective, and those steps are what Y12 (evaluation awareness: the model recognising it is being tested) and Y14 (the agent evaluation frontier) go on to chase.

The one-line takeaway: what you train is behaviour and what grows is an objective; objectives that look identical during training will pick the moment after deployment to part ways with you.