Regression to the Mean

Last year's leader has fallen this year. The easiest reading is that it must have got complacent, slackened off, or been tampered with. There is a plainer possibility that gets overlooked more often: it fell for no reason at all, because the mathematics does that by itself. The phenomenon is regression to the mean, one of the most badly misread statistical facts there is, and misreading it makes you see pure luck fluctuation as causation, as conspiracy, as management failure. The discipline is simple: before blaming a leader's fall on reactivity or Goodhart, subtract the regression layer, and only what is left is up for explanation. A measure fails once it becomes the target everyone charges at (B01). Regression to the mean looks a lot like that kind of measure failure. It is really just a false alarm, and you have to recognize and remove it before you start misdiagnosing Goodhart everywhere.

The phenomenon was first recorded by Galton in 1886: unusually tall parents tend to have children who are less tall and fall back toward the average, which he called regression towards mediocrity. The mechanism is simple and not mysterious once said out loud. Any noisy measurement produces a result equal to the true level plus a bit of luck: the time you scored unusually well, your true level was probably decent and a bit of good luck was stacked on top; whoever reaches the top of the leaderboard usually has real ability plus a lucky break. And luck does not repeat itself to order. So when you measure again, the true level is still there but that extra luck has dispersed, and the score naturally falls back toward where the true level belongs. The more extreme the result, the more luck was mixed into it and the more obvious the fall. Nobody slackened off, nobody cheated, luck just stopped helping.

Kahneman keeps hammering this blind spot in Thinking, Fast and Slow, because it is so easily read as causation. A typical scene: praise someone for performing well and they often do worse next time; criticize someone for performing badly and they often do better next time. So people conclude that criticism works better than praise. In fact both sides are just regression: they were praised because that time was unusually good (with luck in it) and criticized because that time was unusually bad (with bad luck in it), and next time each simply returned to their own normal, with little to do with praise or criticism. History has an even bigger blunder. In 1933 the statistician Secrist collected vast amounts of company data, found that the most profitable firms saw profits fall back a few years later while the worst recovered, and wrote a thick book on that basis reaching a pessimistic economic conclusion: business was heading for the triumph of mediocrity, with the strong eventually levelled. The whole book was really just mistaking regression to the mean for a genuine economic law. It later became a famous cautionary tale in statistics: whenever you follow a group selected on extreme performance forward in time, you will see it regress, and that has nothing to do with mediocrity.

Since the fall is the default setting of the mathematics, how do you avoid being fooled by it? Two ways, neither mysterious. First, set up a control: instead of only watching the leader's own two scores, look at where the middling subjects went over the same period, subtract the natural movement of the whole field, and only what remains can be a real signal. Second, discount extreme estimates, which statistics calls shrinking toward the prior: do not fully believe an absurdly high score, pull it back toward the overall average a little before using it, which Smith and Winkler call disciplined skepticism. Both have the same core: admit that extreme values have luck mixed in and squeeze that luck out first.

This trap does the most damage in organizations and selection. Harrison formally moved it into organizational settings in 1995, and strung three things onto one line in the process: regression to the mean, expectation inflation and the winner's curse. Poach a star into the company on a high current score and your expectations of them get inflated by that score. That is expectation inflation. Once expectations are inflated, next period they will most likely regress to the mean, and "the star declined the moment they arrived" gets misdiagnosed as a management failure or a bad cultural fit. The truth is the winner's curse: what got selected in the first place was simply a lucky high point. The same logic has a household version called the cover jinx. In 1987 Sports Illustrated put two baseball outfielders on the cover of its baseball preview issue and predicted they would win the championship, and their performance that year was ordinary, so the idea that "appearing on the cover brings bad luck" spread. The truth is regression again: a team or a player gets on the cover precisely because they have just produced an extreme peak, and after a peak a fall is due, with the cover merely photographed at the top. There is no curse, only our habit of writing a causal story for regression.

One open question: since a leader's fall contains both pure regression and possibly genuine reactivity or Goodhart, how do you separate the two and apportion them? No framework currently models reactivity and corrects selection bias at the same time so as to split a fall cleanly into "the luck tide going out" and "the measured party really coasting." That separation problem is the most-missing piece in research on eval degradation.

The one-line takeaway: a leader's fall is usually the tide of luck going out rather than a conspiracy; before pinning it on Goodhart, subtract the regression layer completely.