HKU + Meituan's UniClawBench stumps top agents below 50% pass

Every frontier proactive agent tested against a new benchmark out of HKU MMLab and Meituan failed more than half the time, and the numbers are worth staring at before believing any current 'agent that runs your desktop' pitch. In the UniClawBench paper on Hugging Face, the authors report Claude Opus-4.8 leading the leaderboard with a 0.475 pass rate and GPT-5.4 following at 0.407, with all ten state-of-the-art models evaluated landing strictly below 50%.

The setup is what makes the finding load-bearing rather than another leaderboard shrug. Instead of static answer keys inside a sandbox, each of the 400 bilingual English and Chinese tasks runs inside a fresh Docker container with a real browser, terminal and local file system, and gets graded step-by-step against hidden checkpoint rubrics by a supervisor agent. A separate user-simulator agent then feeds up to two follow-up turns of natural-language feedback, sanitised so it cannot leak the grading criteria to the executor. On a 50-trajectory reliability check, the automatic pass/fail decision agreed with a majority vote of three human experts 92.0% of the time.

The capability decomposition is where the diagnostic value lives. Tasks are sliced along Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination, so a failure can be pinned to a specific weakness rather than a scenario label. The empirical read from the authors is that current agents are already reasonably capable of local tool operation and information seeking, but 'still struggle with long-horizon memory, multimodal grounding, and cross-application coordination'. They also flag a 'halfway failure' pattern where checkpoint scores are high but final pass rates lag, meaning models make partial progress and then trip on irrecoverable errors deep into a run.

The more surprising result is buried in the cross-framework table. When the same three models are rerun across OpenClaw, Nanobot, and EDICT, the paper concludes that 'framework choice exerts a stronger influence than model choice'. Nanobot uses roughly half the input tokens of OpenClaw for GPT-5.4 (0.57M vs 1.15M average per task) but pays for that efficiency with lower pass rates on tasks requiring long evidence chains, a concrete illustration of the token-vs-completeness trade in agent scaffolds.

The honest caveats: the benchmark is brand new, the reliability audit covered only 50 trajectories, tasks average 17.4 minutes on modest Intel Core i7-13700 hardware, and the paper does not break out API dollar cost per run. What the reporting does not give you is a per-capability agreement rate for the supervisor or guidance on which framework a specific enterprise workload should standardise on. The direction is worth watching anyway, because a benchmark that credibly separates 'the model is weak' from 'the harness is weak' is the kind of tool teams shipping real agents will actually reach for.