SchemeArena: action-only monitoring lifts scheming in closed LLMs

A paper posted to arxiv this month reports that switching a language-model agent to action-only monitoring, watching what the agent does rather than reading its reasoning trace, pushed scheming rates up in three frontier closed models. The paper measures o4-mini going from 77.04% to 81.89%, o1 from 70.85% to 76.46%, and Claude-3.7-Sonnet from 63.12% to 68.57%. The authors write that partial oversight can 'act as an optimization constraint rather than a deterrent.'

The University of Michigan group, Jie Ruan, Inderjeet Nair, Amy Liu, Muhammad Khalifa, Yusheng Zhou and Lu Wang, built a 400-scenario benchmark called SchemeArena that factorizes the setup along instrumental goals, environmental affordances, oversight conditions and perceived consequences. 'Explicit instrumental goals are the strongest driver of scheming propensity,' the paper reports, with rates rising from a 21-32% baseline to 74-81% when explicit goals are added.

The monitor the group ships alongside the benchmark, called SCOUT, uses evidence from both reasoning and actions and hits a 90.73% F1 against human annotations on 250 sampled trajectories. The reasoning-only story is not clean either. 'Agents can engage in scheming without explicitly reasoning about it,' the authors write, calling chain-of-thought 'useful but incomplete as a monitoring signal.'