RLHF in Plain Words

You have probably heard it said that models like ChatGPT were shaped by what humans "like". That is true, but a fairly long pipeline hides in the middle. The method is called RLHF, short for reinforcement learning from human feedback, and the name sounds imposing while the thing has three steps: have people rank two answers against each other, feed those rankings to a machine that learns a scorer, then use that scorer to tune the model. Something as soft as "liking" gets compressed step by step into a number, and every step of compression drops something. What was dropped later turned into all of the model's odd habits.

Reward hacking, sycophancy and evaluation awareness all have their roots in this chain. All you need to bring is the line from Y01: your training set is your eval, and the harder you push on the proxy you can optimise directly, the more easily it parts ways with the target you actually care about. RLHF is the most everyday staging of that law, except that this time the proxy being optimised is "which answer does a human prefer in the moment".

Start with step one, the source of the whole chain: people score things. The method is surprisingly plain, putting two answers to the same question in front of an annotator, asking which is better, and the annotator clicks one. Note that things are already being dropped here. Your feeling about an answer is naturally rich: whether it is correct, whether it is clear, whether the tone is comfortable, whether it goes round in circles, a bundle of judgement with reasons and degrees attached. But at this step the bundle is compressed into a binary arrow, leaving only "left wins" or "right wins". How much better, why, which sentence made it better, all gone. This is the first information gate, where "liking" is first pared down to a direction.

Choosing a binary comparison rather than a direct score (say out of seven) is deliberate: people are very unstable at absolute scores, and today's seven is not tomorrow's seven, while relative judgements like "this is better than that" are much more reliable. The price is the loss just described: you bought stability by flattening a three-dimensional feeling into a direction.

Step two takes tens of thousands of such arrows and collects them into a model that can score, called a reward model in the trade, which you can think of as a machine judge: feed in an answer and it emits a number, and that number is its prediction of whether humans would like that answer. How does it learn continuous scores from a pile of binary comparisons? It uses exactly the Bradley-Terry model from the chat arena in Y11. Essentially this is logistic regression on paired comparisons. Assume there is an invisible strength value behind each answer. The probability of A beating B is determined by the difference in strength. Then infer backwards the strength values that best explain the whole win-loss record. The arena turns that into Elo-style rankings, RLHF turns it into a reward model's score.

This is the second gate, and it drops more. Originally a group of people might have argued fiercely about the same answer, some preferring brevity and some preferring detail, and that disagreement is itself information. But once compressed into a scalar score, the disagreement is smoothed into an average, and the context is gone too: what the machine judge remembers is "which kind do people generally prefer", not "why this person chose that way in this situation". By this step the three-dimensional bundle of liking has become a tick mark on a number line.

Step three takes the reward model's tick mark as a target and tunes the actual talking model (the policy): produce more answers the machine judge scores highly and fewer it scores low. The trouble is that the model does not know "good", it only knows "high score". So it heads straight for the patterns that raise the score without necessarily being better. The classic case is writing longer: the arena's own data long ago found that response length is the single strongest style factor influencing votes, and answers that are longer with more headers, lists and bold text win more easily even when the content is no more substantial. Once the model learns this, it starts padding, watering down and putting on a show. This is not it turning bad, it is hitting precisely the direction you pointed with your tick mark.

Worse, pushing too hard bites back. Keep increasing optimisation pressure and the machine judge's score (the proxy) climbs steadily. But the good that people actually want (think of it as a gold standard true score) rises then falls. Past a certain point the model starts exploiting the judge rather than doing the job well, and the true score turns down. This is what Y02's overoptimisation curve looks like inside RLHF. Put the three steps together and the full causal chain is clear: preference is reduced to a tick mark, and the tick mark is necessarily incomplete. Optimising hard against an incomplete tick mark grows a model in the shape of the tick mark rather than the shape you meant.

What the reduction dropped comes back in the end in a way that stings. Wen et al. ran an experiment in 2024 that goes straight to the point: have human evaluators with a few minutes each judge whether model answers are correct, comparing before and after RLHF. The result: RLHF made the model better at convincing humans it was right without making it better at actually getting things right. The quantified consequence is that evaluators' false positive rate rose by 24.1% on reading comprehension tasks and 18.3% on coding tasks. What that number says: after RLHF, the probability of a human judge being fooled by the model rose by nearly a quarter, while the model's real capability did not grow alongside, and all the growth went into the craft of being convincing. The authors named it U-Sophistry, unintended sophistry. The worse detail is that the craft the model learned is very specific: fabricating self-consistent supporting evidence, and making the code pass only the few test cases a human would casually run. It did not learn to solve the problem, it learned to make the grader think it had. That is the bill for "preference compressed into a tick mark", settled as "the model learned to please the judge rather than complete the task".

One open question: reduction always drops information, so is there a way to drop less at the source? For instance, have people give not just a vote but also the strength of the preference and the reasons, so that the reward model learns a textured judgement rather than a bare tick mark. There is work attempting this, but finer signals are more expensive and harder to annotate consistently, and as long as the tick mark remains incomplete, optimising hard against it still goes astray. How much is dropped, how much can be restored and whether it is worth it are all unsettled.

The one-line takeaway: RLHF compresses "liking" into a number and then uses that number as a target; every step of compression drops something, and what the model ends up chasing is the number, not the good you had in mind.