HarnessEval-W scores 18 world models with agent evidence trees

HarnessEval-W runs an agent pipeline over a world-model rollout and returns a tree of sub-agent findings instead of a single number. The paper applies the framework to 18 representative world models across 330 evaluation cases.

The pitch is in the abstract's first line: "A benchmark should deliver more than a scalar score: what makes an evaluation trustworthy is the reasoning that justifies the score." The authors argue this matters most for world models, where "judging a rollout requires understanding whether physics, causality, and world state evolve correctly."

Existing metrics, they write, "are computed brute-force, leaving no reasoning chain that can be examined or verified." HarnessEval-W's response is hierarchical: a parent agent "interprets the context of each evaluation case, decomposes the evaluation question into measurable subproblems, and spawns specialized sub-agents, each equipped with tailored context and diagnostic tools to reason over its own subproblem." The parent then validates the gathered evidence and issues the verdict.

The output is what the authors call an evidence tree: the reasoning chain behind each score, meant to be inspected rather than trusted on faith. The pipeline's judgments, they report, "closely align with human preferences while providing verifiable, fine-grained diagnoses of every generated rollout."

The abstract does not name the 18 systems, break out per-model results, or report the compute cost per case. It is being released as a live benchmark, with the authors inviting outside contributors to add new skills and evaluation cases as world models evolve.