ACL 2024 Paper: First-Token LLM Evals Diverge From Text 60%+

There is a quiet way to overstate what your model can actually do, and a Findings of ACL 2024 paper argues most of the field has been doing it. When you evaluate an instruction-tuned model on a multiple-choice benchmark, the common shortcut is to rank the candidate answers by the log probability of the very first token the model would emit. It is fast and deterministic, and it is what a lot of leaderboards run on. The problem, as the ACL Anthology paper puts it plainly, is that "first-token evaluation is not indicative of model behaviour when interacting with users."

The number the authors land on is uncomfortable: mismatch rates over 60% across every dimension they measured, including which option the model actually picks, its refusal rate, its choice distribution, and its robustness under prompt perturbation. The models most affected are the ones we would most like to trust in production, the ones "heavily fine-tuned on conversational or safety data." Instruction-tuning trains models to open with a preamble like "Sure" or with a refusal, neither of which is what the first token of a raw log-probability rank would suggest.

Why it matters if you are not writing eval harnesses yourself: a lot of the safety and capability numbers you see quoted in blog posts and procurement decks are downstream of exactly this method. If the ranked first token disagrees with the text answer more than half the time, then the "score" and the shipped behaviour are two different objects. The authors also show that clamping down on the prompt, forcing the model to begin with an option letter or an example template, does not close the gap.

The honest caveat is that the abstract does not name which models were tested, which benchmarks were used, or which direction the mismatch systematically runs, so how much to discount any given leaderboard number is genuinely unclear from this alone. What the reporting does not give you is a fix, only a warning to inspect the text output as well.

The forward-looking read is that eval frameworks which add a text-output pass, and teams that treat first-token log-probabilities as one signal rather than the signal, get a cleaner picture of what their users will actually see.