Honest Signals & Trade-offs

How do you tell whether an agent is genuinely capable, or just putting on a show for the test? An unexpected criterion comes from biologists who study peacock tails: what makes a signal credible is not that it is expensive, but that it is expensive for liars. Once you see that distinction, you know which kind of criterion to hunt for: the kind a bad system cannot fake no matter how hard it tries, and a good system produces without effort.

You met Goodhart's law in B01: once a metric becomes the target everyone pushes on, it stops being a good metric. The principal-agent problem in B12 said the same thing another way: you want a good agent, but the optimizer only wants the eval to light up green. If metrics always get gamed, is there a kind of signal that is inherently harder to fake?

Biology has been chewing on this for a while. A peacock drags around a huge, cumbersome tail that ought to get it eaten first. Zahavi said in 1975 that the very burden is what makes it credible. How does that work? Only a genuinely healthy peacock can carry that much burden and still thrive; a sickly one cannot. That gap between affording it and not affording it is what makes the tail reliable evidence of health. The idea is called the handicap principle, and its reputation has risen and fallen three times.

Three acts, one sentence: honesty does not come from being expensive, it comes from being expensive for liars. The same tail is affordable to a healthy peacock and unaffordable to a sick one. That cost gap is what matters. Absolute expense turns out to be unnecessary. Textbooks usually stop at the first act, the "costlier means more credible" version. The field has already moved on.

Follow that sentence and there is an easier route. Economists have a term, cheap talk, meaning words that cost nothing to say. It proves something useful: if the speaker and the listener have aligned interests, telling the truth costs nothing and cheap talk is already credible. Flip it around, and the greater the conflict of interest, the more you need expensive signals to hold lies down. In one line: the cost of maintaining honesty is proportional to the size of the conflict of interest.

An everyday example. You ask a teammate whether a piece of code has bugs. He is on your team, and a bug in it lands on his own plate, so he has no reason to lie. His offhand answer is credible. But if he is a contractor who gets paid sooner by saying "looks fine", you now have to add acceptance testing and accountability, which means adding cost. Same question. How much cost you need depends entirely on how far apart your interests are.

That gives you a design order that runs against intuition: do not start by cranking the bar as high as it goes. First find ways to align the interests of whoever designs the test and the agent being tested, and bring the conflict down. Then decide whether cost is still needed. Even when it is, you do not need to go to the ceiling. Attaching a small but certain penalty to false reporting (some probability of being audited, accountability after the fact) pulls honesty back a long way.

The toughest kind of criterion drops cost altogether and leans on physics. Some signals simply cannot be produced by a low-quality party. A small bird cannot make the deep call of a large bird, because pitch is pinned down by body size. The signal takes no effort and cannot be faked. These are called index signals. In eval work they are the most valuable thing you have: a program that actually runs, an experimental result that reproduces, real consequences left behind after a multi-step operation. All of these are determined by the structure of the task, and a weak model physically cannot produce them. They beat any "very hard" scoring scheme. G05 pushes this into formal mechanism design, and the workbench is in C03.

This is not theory. A batch of coding benchmarks fell down exactly here: models scored very high, and it later turned out they were mostly reciting, word for word, the fix patches humans had already written. Memory, not reasoning. If the criterion had been aimed at index-type things instead, such as the real difference in program behavior before and after the change, or whether it passes a set of tests it has never seen, then reciting answers would earn nothing. A bad system cannot fake "actually runs". That is what an index signal is worth.

To check whether a criterion resists faking, ask three things in order:

• Interests first: do the test designer and the agent under test have aligned interests? Can you make the designer someone who has to use this agent for real work? The smaller the conflict, the less work everything downstream takes.
• Then look for structure: does this task contain anything a weak model physically cannot do? Programs that run, results that reproduce, operations that leave real traces. Use those as criteria first.
• Then compute the cost gap: how much more effort does a bad system have to spend to produce a good-looking result? If bad and good systems find it equally easy, or equally hard, the criterion is useless. What you want is: very costly for the bad system, free for the good one.

One inoculation in advance: any criterion, once everyone has figured it out and started optimizing against it, slowly stops working. Same logic as credential inflation. A university degree proved something while it was rare, and stopped being worth much once everyone had one. The tested party converges on the criterion and discrimination fades. A public coding benchmark usually goes this way: in year one it separates models, and two or three years later everyone is grinding on it, scores bunch up at the top, and it is finished. So anti-gaming is not "design one good criterion and you are done". Better to give every criterion a service life, estimate how long until it is figured out, and refresh on schedule. This connects to the dynamic refresh in G03 and the metric rotation in G04.

One open question: if honesty rests on difference rather than expense, then in an artificial scoring system, should we be reproducing the difference in cost or the difference in benefit? Biology has that structural unfakeability where a small bird simply cannot produce a low note. Is there anything in an artificial eval that corresponds to it? This is the key gap left open by the D7 report.

The one-line takeaway: a good criterion is not the hardest one, it is the one a bad system cannot fake and a good system produces without effort.