Found first: a primary source the press has not covered yet.
A new benchmark tests whether LLM agents can reason over rules distributed across multiple files, the way real compliance and policy work actually looks. Across 11 frontier models, end-to-end task completion ranges from 26.97% to 55.29%. The paper, RuleTrek: Benchmarking Agentic Rule Reasoning in Complex Scenarios, is on arXiv.
What the source says
RuleTrek places rules inside an interactive file-system workspace where they are distributed across multiple documents rather than provided inline. The benchmark spans six domains: taxation, PostgreSQL, cybersecurity, software policy, procurement, and game rules. Three capabilities are scored separately: Rule Identification, where the best model reached 62.47%; Rule Deduction, where the best reached 60.43%; and End-to-End Rule Reasoning, where all 11 models fell in the 26.97% to 55.29% range. The pipeline extracts structured rule graphs and generates rule-specific scenarios through incremental composition to ensure consistency and solvability. The 13 authors include Chenhao Yuan, Bohan Yu, Jun Zhao, and Kang Liu; code and dataset are public on GitHub.
Why it matters
Existing benchmarks hand rules to models inline, which bypasses the retrieval and cross-document reasoning step that real-world agent deployments require. Tax codes, procurement guidelines, and security policies live in separate files, and RuleTrek is built around that fact. The gap between Rule Identification scores in the low 60s and end-to-end scores that cap below 56% suggests failures are compounding: a model that locates a rule does not reliably apply it through to a final answer. That drop matters most for the enterprise use cases, such as automated compliance checks and policy-driven procurement agents, where partial rule application produces wrong answers rather than graceful failures.