RM Overoptimization

What happens if you take a scoring standard that is roughly right and optimise an AI hard against it? The answer is not "the effect is discounted", it is "better first, then worse", and the turning point has experimental data and a functional form. Plenty of people treat Goodhart's law as an aphorism. This curve turns it into something you can plot on graph paper.

Today's chat models mostly go through a step before shipping called RLHF (reinforcement learning from human feedback): have the model generate several answers to the same question, have people pick the better one, then use "getting picked" as the reward to adjust the model. The trouble is that people are expensive. Training needs hundreds of thousands of comparisons, and you cannot ask a real person every time, so engineers first take a batch of human choices and train a reward model: a scorer that has learned to imitate human scoring. From then on training asks it, not people.

Note the substitution in that step: the scorer is only an approximation of human preference, a proxy. Optimising hard against an approximate standard is precisely the standard setting of reward hacking in Y04, where the question was whether it happens at all. Here we push one step further and ask about quantity: after how much force do the bad things start, and how fast.

In 2023 OpenAI's Gao, Schulman and Hilton designed a neat experiment to measure it. Since real people are too expensive, they used a very large "gold standard" reward model to play the human (call it gold), trained a smaller proxy scorer from gold's scores (call it proxy), then optimised the model only against proxy while quietly keeping score with gold on the side. Gold is, after all, a model, not a real person: it can be called any time, scoring without limit. That makes "true quality" readable an unlimited number of times, and optimisation force can be pushed to depths no experiment with real humans could reach.

The result is a parabola: proxy's score climbs steadily, gold's score rises then falls. Translated: from the proxy's point of view the model keeps improving; from the gold standard's point of view the model starts getting worse past a certain point. When all you hold is the proxy, you have no idea that you have already gone past the peak.

First the horizontal axis. The paper measures "optimisation strength" with KL distance: how far the model has been pushed from its initial version. Plainly: how much it has been remade. The harder the remaking, the further the model is from its factory state and the larger the KL. A KL budget is simply a cap on the extent of remaking.

Dragging the slider shows two things. First, the greater the mismatch between proxy and true objective (the less the scorer resembles a real person), the earlier true quality peaks and the harder it drops. Second, adding a KL constraint lets you stop near the peak, which is the standard RLHF countermeasure to overoptimisation: not refusing to optimise, but limiting how deep the optimisation goes.

The paper also fits a functional form for the curve, with two sets of coefficients corresponding to "how fast it rises at first" and "how hard it falls once past the top", and the coefficients vary smoothly with scorer size. That yields an important corollary: make the scorer bigger and the peak is higher and the turn later, but the shape of the parabola does not change. The shape not changing means enlarging the scorer only shifts the curve's position, it cannot move the bend itself. Overoptimisation is structural, not an engineering blemish caused by an undersized scorer. Enlarging the proxy can delay the turn, it cannot cancel it.

After 2024 this one chart grew into a family, with three findings each adding a cut.

First: changing algorithm does not dodge it. A newer class of training methods (DPO and other "direct alignment algorithms") skips training a scorer entirely and learns straight from the human choice records. That sounds like routing around the proxy, and the experimental answer is that it does not: the same parabola shows up, and because a layer of buffering is gone, the degradation is harder to control in form. As long as what you optimise is an approximate good, the parabola is there, independent of algorithm family.

Second: every mitigation delays without cancelling. Training several scorers and having them vote (an ensemble), handling disagreement conservatively, clearly delays the turn; averaging several scorers at the parameter level (WARM) is cheaper in compute and steadier still; using an information bottleneck so the scorer only sees features genuinely relevant to preference (InfoRM) helps too. But the title of one mitigation paper says the shared ceiling out loud: mitigate but do not eliminate. And ensembles have a hidden limit: if every scorer learned the same bias from the same data (say "longer answers are better"), the same trick fools all of them at once. A panel of reviewers sharing a bias equals one reviewer.

Third: multiple rounds accumulate. Run RLHF as an iterated loop (retrain the scorer on data generated by the previous round's model, then optimise the next round) and overoptimisation stacks across rounds and becomes more sensitive to hyperparameters. The longer one proxy is overdrawn, the bigger the hole.

Two theorems on the theory side cap the chart. Skalse et al. proved that unless a scoring standard is the useless one that gives the same score no matter what you do, there necessarily exists a way to game it. That is a theorem, not a lesson learned, which means "design a completely cheat-proof reward" is mathematically impossible, and all engineering can do is raise the cost of gaming. Zhuang and Hadfield-Menell proved that when your score covers only some of the dimensions you care about, optimising it without bound necessarily sacrifices the uncovered dimensions arbitrarily badly. Together: a proxy can always be played, and playing it to the end always bites back. Both theorems describe a limiting case (a loophole exists, unbounded optimisation must sacrifice), and neither says what the path looks like between a loophole's first appearance and the eventual backlash. Gao's parabola is the empirical interpolation between those two theorems.

This curve is the prototype for two lines at once. On the machine side, the sycophancy of Y08 is the concrete content of what lies past the turn: what the error term of an overoptimised human preference proxy contains is precisely "say what you want to hear". On the defence side, the prescription is read straight off the curve: KL budgets and early stopping are the engineering of "stop where the true objective peaks"; ensembles, weight averaging and information bottlenecks all raise the cost of harvesting the error term, and G05 turns "raise the cost of cheating" into a general principle.

It also pairs with "reusing the same exam wears the exam out" (Y02): that side is a mild chronic overdraft, this side is the acute overdraft of directed optimisation against a single proxy. What separates the two ends is how concentrated the optimisation pressure is.

One open question: every mitigation has the same shape, delaying the turn, while identifying the turning point still relies on empirical early stopping, because there is no gold score to read in deployment. Is there a provably optimal stopping rule that can identify the peak without knowing the true objective? That is the open problem left in research/deep/D5 §2, and also the question every engineer filtering outputs with an AI judge answers by hand every day.

The one-line takeaway: for an imperfect scoring standard there is an optimal amount of optimisation; past that point, the higher the score the worse the thing, and anyone watching only the score cannot tell.