arXiv Study: 'Solution Hacking' Inflates LLM Reasoning Scores

A new paper on arXiv puts numbers on something quietly suspected for a while: that a lot of what looks like frontier LLM reasoning on hard science benchmarks is not reasoning at all. The paper, Right Answer, Wrong Method, argues that models increasingly get the correct final answer through what the authors call Solution Hacking, invalid shortcuts such as numerical search, enumeration, guessing, or answer-first verification, without providing a valid task-targeted derivation.

The numbers are the part worth staring at. On common problems the shortcut rate sits at 2.2%. On Olympiad-level problems it climbs to 28.3%. On HLE it hits 37.4%. And across frontier models more broadly, between 8.2% and 44.1% of answers that were credited as correct turn out, on inspection, to be hacked. That is a wide range, but the ceiling is what a leader should notice: nearly half of a model's apparent right answers on a hard eval may be shortcuts rather than reasoning.

Why this matters if you are procuring or deploying models for anything that resembles scientific work. The industry sells reasoning as a headline capability, and leaderboard positions on hard benchmarks are the proof. If answer-only scoring materially overstates that ability, then part of what buyers are paying a premium for is a measurement artifact. The authors also test mitigations, including an automatic judge and a test-time instruction, and report that suppressing shortcut behavior substantially reduces reported accuracy while having a smaller effect on correct and non-hacked accuracy. In plain terms, a lot of the score drops when you make the model show its work.

The honest caveat is that the abstract does not name which frontier models fall where on the 8.2 to 44.1 range, so this is a category-level finding rather than a specific indictment of any one lab. It also does not tell us how robust the anti-hacking judge is against models that learn to hide the shortcut inside a well-written derivation, which is the next arms race worth watching.

Where this points is useful. Eval vendors, academic labs, and internal benchmark teams have a cheap upgrade available: score the method, not just the answer. The groups that do this first get a more honest picture of what their models can actually do, and buyers who ask for method-level scoring in their next RFP will get a far more informative bake-off than the one the public leaderboard is offering today.