Fuse benchmark: 12 LLMs trail humans on user-mediated advice
TL;DR
- On Fuse, the human majority baseline reaches 88% correctness and 89.8 MSR; no model among 12 frontier LLMs tested exceeded 83.7 MSR.
- Under biased user framing, the average model gap is 7.7 points versus a 3.6 drop for humans, with bias-gap variation of 6.9 to 12.5 across models.
- Model performance climbs through turn 4 of conversation, then plateaus or degrades as clarification breaks down and bias compounds.
On Fuse, a new benchmark for user-mediated social reasoning from Amir Taubenfeld, Zorik Gekhman and colleagues, the human majority baseline scores 88% correctness and 89.8 MSR. No model, out of 12 frontier LLMs tested, exceeded 83.7 MSR.
The setup: a target agent with a hidden motive interacts with other agents, one of which represents the user, who then consults the evaluated assistant to infer that motive. Ground truth is fixed by construction. The authors validated the simulation with a human study of 24k annotations and released a dataset of 21k examples.
What breaks the models is framing. The paper reports that "LLMs exhibit systematic sensitivity to biased user framing," and that "the average model gap (7.7) is more than double the drop in human baseline (3.6), suggesting that the degradation is not driven by information loss alone." Bias-gap variation across the tested models runs from 6.9 to 12.5 points.
Longer conversations do not always help. Performance climbs through turn 4 as users provide clarifying evidence, "then plateaus or degrades due to ineffective clarification and compounding bias." Models can require more details than humans to reach a correct prediction.
Originally reported by paper
Read the original article →Original headline: Fuse Benchmark: All 12 Frontier LLMs Fall Below Human Baseline on Social Advice Filtered Through Biased User Narratives