DocOps benchmark finds frontier agents falter on document tasks

A new paper on arXiv titled DocOps takes aim at one of the most commercially loaded agent use cases, document manipulation, and reports that the frontier configurations the authors tested still fall over on the harder end of the workload. Posted July 22, 2026, the paper introduces what the authors call a deterministically verifiable evaluation framework, built on a hierarchical taxonomy that decomposes document operations into atomic dimensions and escalating workflow complexities.

The headline claim, in the authors' own words, is that 'even the most advanced frontier configurations still exhibit profound limitations when handling highly coupled, long-range tasks.' That is a qualitative verdict rather than a leaderboard number, and it is aimed squarely at a workload category many enterprise vendors are actively selling into. If it holds up under wider testing, the practical read is that the safest place to deploy today's document agents is on short, low-coupling tasks, not on the multi-step restructure-this-quarterly-report work that tends to headline vendor demos.

The more useful part of the paper, at least for practitioners, is the three failure modes the authors say they observed: long-term state tracking collapse, shallow semantic verification, and destructive editing of structural metadata. Those are named, diagnosable behaviours, which means a team building or buying document automation can look for them in their own evaluations rather than only in aggregate scores.

The honest caveat is that the arXiv abstract does not disclose which specific closed and open models were tested, what the pass rates were, or how large the task set is, only that the study covers 'representative closed- and open-source models across various agentic harnesses.' Those specifics would matter, particularly the model list and the version dates, because agent capability on this class of task has been a moving target through 2026. What the paper does contribute right now is a shared vocabulary for the failure modes and a benchmark whose verification is deterministic rather than judge-model based, which is the more durable of the two contributions.