Lucas Critique

Goodhart's law has a twin brother: the Lucas critique, put forward in 1976 by the economist Robert Lucas (B01 planted this thread at the end). The plain version: you find a stable regularity in historical data and want to use it as a lever. But the regularity has people's expectations about the old rules baked into it: the moment you pull, what you're actually moving is that very expectation. The expectation shifts, behaviour follows, and the regularity dissolves on the spot. The regularity you used to predict stops working because you used it. This explains why relationships computed from old data cannot hold up under new policy, and it explains why a benchmark (a public question set giving every model the same problems and scoring) starts failing as soon as it becomes a target.

Start with the textbook accident. Data from the 1960s showed an apparently stable trade-off curve between inflation and unemployment, called the Phillips curve: a bit more inflation, a bit less unemployment. Policymakers read a tempting operational implication out of it: tolerate steady moderate inflation and you can buy steady low unemployment. But the slope of that curve itself had the public's expectations about the old policy environment baked in. Once the government started systematically exploiting it, relying on sustained moderate inflation to buy low unemployment, the public saw the manoeuvre plainly. They saw through the routine, wage and price setting changed accordingly, and the trade-off vanished. The 1970s ended in stagflation: high inflation and high unemployment together, with the bargain the curve promised simply not existing. The regularity was not wrong. What was wrong was treating a passively observed regularity as a lever you could actively push. The curve was not falsified, it was used to death. That sentence could be carved, unchanged, onto every saturated benchmark.

Lucas compressed the logic behind the accident into what he called a "single syllogism." In plain terms: first, the parameters of an econometric model (the system of equations estimated from historical data) come from millions of ordinary people making optimal decisions under the old policy; second, people's optimal decisions vary systematically with changes in the rules; third, therefore any change in policy will systematically alter the structure of econometric models. Not one of the three sentences is spare. Parameters are not natural constants. They are the reduced-form shadow of behaviour under the old rules, and change the rules and the shadow rearranges. The conclusion follows: using the current econometric model to compare the effects of alternative policies is invalid, no matter how beautifully the model fits history or how accurate its short run forecasts are.

Goodhart's British scene (the failure of monetary targeting covered in B01) slots into that syllogism sentence by sentence, and the two laws annotate each other inside the same stretch of history. The stable relationship between monetary aggregates and the economy was the reduced-form shadow of banks and firms behaving optimally under the old controls. The central bank rewrote the policy rule twice (loosening controls in 1971, hard controls at the end of 1973), agents reoptimized, banks invented new liabilities outside the controlled definition, and the relationship between the targeted aggregate and the economy came apart at once. The only difference is where the narrator stands. Lucas stands on the modeller's side and says "your parameters will drift." Goodhart stands on the central bank's side and says "your target will collapse." They are describing the same accident.

Lucas's constructive claim is often ignored: the way out of the critique is to build models on deep structural parameters, that is on preferences, technology and resource constraints, the quantities that do not change with policy (the term of art is policy-invariant), and to eject the surface parameters that drift with policy from the extrapolation chain. In one line: a prediction that will not be destroyed by reactivity must be built on causal structure, not surface correlation. Fifty years later that judgment acquired a formal counterpart in machine learning: Y10 (strategic classification) proves that gaming-proof prediction requires causal modelling, and Y09 (performative prediction) writes "the prediction itself changes the distribution" as mathematics.

Its relation to Goodhart's law is closer than "similar." Chrystal and Mizen's 2003 ruling reads: it can be argued that Goodhart's law and the Lucas critique are essentially the same thing, and if so, Lucas almost certainly said it first. Their division of labour lies in the register. The Lucas critique changed the methodology of macroeconometrics (models must be built on deep parameters); Goodhart's law changed the design of monetary policy (money targeting left the stage, inflation targeting took it). As for "who was first," it depends on whether you count reading at a conference or formal publication as the debut: Goodhart read his paper in Sydney in July 1975, Lucas published formally in 1976, and Lucas's conference draft actually circulated earlier. That priority question is still unresolved.

Placed in the larger family tree, the Lucas critique is the economic-methodology version of the same insight. Ridgway had already reviewed measurement failure in organizational research in 1956 (the prequel covered in B01), Campbell's corruption proposition was already in embryo in 1969 (B02: indicators corrupt the social processes they measure), Goodhart gave the empirical generalization for monetary institutions in 1975, and Lucas gave the microfoundations and a general theorem in 1976: the mechanism is rational expectations (the assumption that people anticipate policy and adjust behaviour accordingly), and the way out is to separate deep parameters from reduced-form ones. Several disciplines hit the same wall one after another, and Lucas is the one who wrote the mechanics of the wall as a theorem.

This distinction has already surfaced once: the detail about banks inventing new liabilities to dodge the controlled definition was not just reoptimizing, it was an extra layer of active evasion. That is also why Goodhart's law can be read, roughly, as the Lucas critique plus gaming avoidance in a monetary setting. Lucas's agents merely reoptimize their own decisions; the banks Goodhart faced also actively invented new liability instruments to get around the controlled definition, and that extra layer is adversarial. The full taxonomy of adversarial mechanisms is in B08 (the four Goodhart types: a vocabulary splitting measure collapse into regressional, extremal, adversarial and causal mechanisms).

The mapping onto evaluation is very direct. For a static benchmark, "score correlates with capability" is a reduced-form parameter estimated on a distribution where nobody was optimizing against it: the training recipe, data mixture and sampling strategy did not treat that benchmark as a target at the time. The moment it is published and treated as a target is equivalent to a policy change. The trainers reoptimize, the score's generating mechanism changes, and the old correlation no longer answers for real capability after deployment. Y02 (adaptive overfitting) and Y03 (benchmark saturation) are two readings of that same parameter-drift curve on the leaderboard.

One open question: what exactly is a deep parameter in the eval setting? Economics treats preferences and technology as the bedrock that policy does not move, but for a model that training itself reshapes, almost no internal quantity is "optimization invariant." Finding eval's deep parameters, or proving they do not exist: which do you bet happens first?

The one-line takeaway: a historical regularity is the shadow of everyone's behaviour under the old rules; change the rules and the shadow moves, so a prediction that survives intervention has to grip something the rules do not move.