Leaderboard Illusion

How much water is in the scores on the world's most influential AI model leaderboard? Swap the red branch's law school rankings story for model leaderboards and the structure of reactivity is unchanged word for word, only the cycle shortens from years to weeks. This is also where R10 (popularity amplifies itself) and Y03 (exam questions get eaten into training sets) meet.

The leaderboard first. Chatbot Arena (later renamed LMArena) has an elegant mechanism: a user asks a question, two anonymous models each answer, the user votes for the better one; a statistical model (Bradley-Terry, essentially computing an overall strength for each competitor from a mass of pairwise comparisons, the same idea as chess ratings) then aggregates millions of matches into a score and rank for each model. Anonymity removes brand bias and real votes stay close to real use, so it was treated for a while as the gold standard leaderboard for "true ability". The gold standard bought attention and adoption, and that same benefit made it the number one optimisation target.

In April 2025 the Cohere team, together with several institutions, published The Leaderboard Illusion, the most systematic dissection of this board to date: an analysis of roughly 2 million matches between January 2024 and April 2025, covering 243 models from 42 vendors. The question to be taken apart is one sentence: of the scores on the board, how much is capability and how much is adaptation to the board itself. The paper's most striking estimate: even a small amount of Arena battle data used for tuning can deliver relative gains of up to 112% on the Arena distribution. Translated: you do not have to make the model better, you only have to learn this board's taste, and the score climbs a lot.

Drag the slider and watch a model that truly ranks 11th get lifted into the top few. There are three main charges, all reproducible by hand in the simulation:

• Best-of-N private testing: a few vendors can privately test many variants before release, publish only the highest scorer, and quietly withdraw the results they dislike. All scores carry a luck component, and picking the maximum from a pile of noisy scores mostly picks the one luck favoured (this is exactly the optimiser's curse mechanism of B13), so the expected rank is systematically inflated. The paper alleges Meta privately tested as many as 27 variants before releasing Llama 4.
• Asymmetric data access: the paper estimates Google and OpenAI received about 19.2% and 20.4% of all battle data respectively, while 83 open-weight models together got only 29.7%. More matches means a steadier rank; poorly performing variants can also be silently delisted without a trace, manufacturing survivorship bias.
• Style points: a model need not be more capable, only better at pleasing voters: longer answers, more lists, an enthusiastic opening, emoji. The score rises, the model is not necessarily better.

The style charge has one naked example. On 5 April 2025 Meta released Llama 4, with a blog post promoting Maverick's high LMArena score (ELO 1417). Within days it emerged that what was on the board was an "experimental version" specially tuned to be conversationally likeable, verbose and fond of piling on emoji, and not the same thing as the official version actually released for people to use. Tech media followed up and the community called it a bait and switch. LMArena apologised and changed its policy, putting the untuned official version back on the board, where it landed below GPT-4o, Claude 3.5 Sonnet and Gemini 1.5 Pro. "Shaping the model into the shape of the board" here is not a metaphor, it is a product decision.

The board's own remedies are worth recording. Back in August 2024 LMArena acknowledged that response length is the single strongest style factor influencing votes, and shipped style control: put style variables like length, number of headers and number of lists into the statistical model, estimate them jointly, then subtract their influence from the quality score. Once applied, models known for verbosity slid down the rankings. A measuring body explicitly modelling "the shape that got optimised into existence" and then subtracting it is the arms race between reactivity and counter-reactivity, with both attack and defence evolving.

Between 2025 and 2026, two new lines of evidence pushed the problem further. First, vote rigging was shown to be quantitatively attackable: Min et al. simulated on 1.7 million real votes and found that pouring votes only into the target model's own matches is very inefficient, taking over ten thousand votes to gain one rank. But this statistical model has a property: any single vote nudges the relative positions of all models, even in a match unrelated to the target model. Exploit that global coupling together with a classifier that identifies the anonymous models, and about twenty thousand votes can lift a target model ten ranks; in a realistic restricted-access setting, a few hundred votes buy five ranks. Second, the judge acquired a business of its own: LMArena incorporated out of its Berkeley academic project, closed a $100 million seed round in May 2025 (at a valuation of about $600 million), and raised a further $150 million Series A in January 2026 at a valuation jumping to $1.7 billion. The most influential model leaderboard of the moment is operated by a company whose core product is that ranking. On top of measurement reactivity there is now a layer of capital interest, and the governance question is handed to Y15.

Two earlier lines converge here. R10's music experiment proved that in a market where you can see other people's choices, a small early popularity advantage snowballs into a huge final gap. A leaderboard is exactly such an artificial market: a "tops the chart on release" headline brings trials and goodwill, goodwill becomes votes, and the Matthew loop closes. Y03 comes in from the question side: vendors with battle data can make their models adapt specifically to this board's question distribution, which is exactly what the 112% figure measures. Popularity amplifying itself and taste being learned in advance are two machines stacked on the same board, so "tops the chart on release" may be strength, or may just be the machines turning.

One open question: Min et al. proved vote rigging is technically feasible, but how much coordinated rigging has actually happened in the public data is unmeasurable by anyone, which is reactivity's dark figure. Once a measurement system's gameability is quantitatively proven, by how much should its readings be discounted, and can the discount itself be estimated? research/deep/D4 §4 also lists "whether a commercialised live leaderboard can still self-correct" as an open problem.

The one-line takeaway: the more a leaderboard matters, the more "making the model good" and "making the board score good" become two different businesses, and the score itself cannot tell you which one you are looking at.