Baidu benchmark: office-suite LLMs cheaper, still trail humans

Baidu's Agent Frontier Team just dropped a useful deflation of the office-agent hype cycle. Their new paper on Hugging Face introduces OmegaUse-OfficeVal, a benchmark of 100 long-horizon tasks pulled from authentic workplace requests across documents, spreadsheets, presentations, and PDFs. The distinguishing feature isn't the task count, it's the economic scaffolding: every task carries a measured human-labor-time annotation (2.32 hours on average, up to 8.35) and a price proxy in dollars (mean $6.86, median $5.11), so you can compare an agent's completion quality against what the same job would cost and take a person.

The scored results are honest. The human baseline reaches 27.79; the best LLM tested, Baidu's own GLM-5.2, gets to 17.91. Qwen3.7-Plus (17.51) and Kimi K2.6 (17.00) trail closely, while DeepSeek-V4-Pro (14.48) and MiniMax M3 (13.82) sit further back. Zero-score task rates run from 38% for Qwen3.7-Plus up to 51% for MiniMax M3, meaning roughly half of MiniMax's outputs failed the usability gate entirely. Humans produced zero-score work only 29% of the time and scored above 50 on 21% of tasks.

Where the story flips is efficiency. Every model was substantially cheaper and, in most cases, faster than the human: Qwen3.7-Plus averaged $0.2152 and 0.193 hours per task versus $6.8560 and 2.324 hours for the human baseline. The paper's own framing is that evaluated LLMs are "substantially cheaper and faster than the human baseline under our scaffold, but still lag behind junior human workers in deliverable quality." The value-weighted metrics also matter for procurement: Qwen3.7-Plus, not the top-average GLM-5.2, captures the highest time- and price-weighted scores, so ranking on averages picks a different winner than ranking on the money.

The honest caveats are worth stating. This is a Baidu-authored benchmark evaluating five Chinese-lab models, with no reported comparisons against GPT, Claude, or Gemini, and the agent scaffold is programmatic file manipulation rather than GUI computer-use, so screen-driven office agents aren't in the picture. The price proxies are recorded in CNY and converted to USD at a 180-day average, so mapping them to Western freelancer rates is left to the reader.

The useful move for anyone building or buying office-suite automation is that the full assets, including task instructions, input files, rubrics, code verifiers, and the per-task economic annotations, are being released, which gives model teams a reproducible target to iterate against and gives buyers a workflow-realistic testbed to run internal pilots on instead of trusting a vendor demo.