PAPAV survey: 3 of 64 agent benchmarks test outcome verification

Only 3 of the 64 agent benchmarks surveyed in a new Hugging Face paper explicitly measure whether an agent can verify its own outcomes, and only 5 test whether it can anticipate what an action will do. The rest grade almost entirely on the last step: did the task get done. Multimodal Embodied Agents: A Unified Capability-centric Perspective from Computer-Use to Robot-Use proposes PAPAV, standing for Perceive, Anticipate, Plan, Act and Verify, as the five functions every multimodal agent runs, then uses that lens to score 10 embodied-agent, 23 computer-use and 31 robotics benchmarks.

"Across 64 benchmarks, explicit evaluation centers on Act; only 5 assess Anticipate and 3 assess Verify, leaving both largely hidden behind task success," the paper reports.

The authors argue the stakes rise when an agent moves off the screen. Five physical constraints (partial observability, limited simulatability, irreversibility, real-time coupling and unverifiable outcomes) leave "fewer choices fixed in advance and less room to reverse errors" than digital environments. End-task success rates can therefore conceal whether an agent predicted the right consequence or caught its own mistake. The finding lands amid a run of embodied-agent coverage on our end, with 166 robotics stories in the last 90 days alongside 515 on agents.

The benchmark list spans computer-use suites like WebArena, GAIA and OSWorld2.0, embodied simulators such as TEACh and RoboCerebra, and robotics evaluations including LIBERO, ManiSkill2 and BEHAVIOR-1K. The paper does not publish per-benchmark scores on the five capabilities.