HVTB paper adds hack detection to Terminal Bench agent tests

A group of researchers has extended a technique for automatically catching AI reward hacks — cases where an agent is 'satisfying a task's checks while violating its intent' — to Terminal Bench, one of the field's most-used coding-agent yardsticks. The preprint by Amit Roth, Ivan Bercovich and Yonathan Efroni calls the result Hack-Verifiable Terminal Bench, or HVTB.

The technique borrows from a methodology called hack-verifiable environments, which works by 'embedding detectable hacks into tasks, allowing reward hacks to be identified automatically and reliably.' That matters, the authors argue, because current detection 'typically relies on human inspection or LLM judges, both of which can be unreliable.'

The paper measures reward-hacking rates across frontier models and probes whether prompts carrying different amounts of hack-specific information change behavior. The design, the authors write, lets them test whether prompting can prevent 'not only known reward-hacking strategies, but also "unknown unknown" exploits that the prompt does not anticipate.' The abstract does not name the models tested or publish per-model rates. Environments and agent traces are posted at the project page.