APort Vault: 79.4% of Level-4 attacks made AI agents pay out

On 1,293 prompts targeting a live payment agent, request rates across fourteen frontier AI models ran from 71.2% to 84.3%, and 809 of those prompts, 62.6%, got a payment request from every model tested. Each request ended in a successful payment to the level's allowlisted recipient.

That result comes from APort Vault, a benchmark posted to arXiv on September 18, 2026 by Uchi Uchibeke, replaying 4,371 human-written attacks against a live payment agent across 14 models from 8 labs and completing 225,964 evaluations.

The paper's central move is to score the same attacks twice: once against the model alone, once with a deterministic pre-action check implementing what the paper calls the Open Agent Passport (OAP) specification. At Levels 2 to 4, transfers to recipients the passport did not permit numbered "140 of 76,842 with the model alone and 0 of 69,297 behind the layer." The 105-against-0 result on 68,970 matched model, prompt and track triples is the like-for-like comparison.

Uchibeke is careful about what zero means. It spans 790 source sessions, "giving a per-session upper bound of 0.38%." And it was not obtained by refusing payments: 25,370 payments executed behind the layer, while the policy denied 187 of the 25,640 transfer calls it evaluated, 148 of them for a forbidden recipient.

The methodology note is unusually blunt for a benchmark abstract. "We report five distinct events per evaluation, because collapsing them is how an agent benchmark produces a number that does not survive review." The 79.4% Level 4 request rate is the model-alone figure; Level 3, a stricter policy tier, drops that to 0.1% before the pre-action check enters the picture.

The paper does not name the 14 models or the 8 labs.