DeepLethe's FDE-Gym: Top Agent Solves 14 of 95 Delivery Tasks

The best agent configuration tested on a new enterprise-delivery benchmark resolved 14 of 95 tasks. Five of the thirteen configurations resolved none at all.

The benchmark, FDE-Gym, was posted on Oct 16 by DeepLethe with co-authors across more than 20 universities and companies, among them Tsinghua, Shanghai Jiao Tong, Beihang, Beijing University of Posts and Telecommunications, Harvard and Stanford. It adapts real customer delivery projects, carried out in China, Singapore and the United States since 2024, into 95 tasks across 20 scenarios and 18 sectors. Industry experts' total contribution 'exceeded 10,000 hours,' and the graders were tested against more than 270 shortcut and bypass strategies before admission.

The scoring is deliberately business-shaped. Zero is the do-nothing baseline. One is an expert-written reference solution. Literal compliance with the sponsor's request is capped at 0.2, and 'a task is considered resolved if its score is at least 0.8 and no incident occurs.' Each run is capped at 200 turns, 150 minutes, or USD 8 of model usage before the delivered system is deployed and graded on held-out production data.

At L3, the setting where the agent sees only the sponsor's original request, Claude Opus 5.5 posts a mean score of 0.56 (95% interval 0.50 to 0.62). Claude Sonnet 5.5 follows at 0.49, GPT-6 Astra at 0.38, GPT-6.1 Sol at 0.37. Information discovery is not the ceiling. 'At L3, leading models obtain 72–85% of the key facts available from stakeholders, yet the delivered systems achieve only 0.60–0.70 of the reference improvement.' Stakeholder contact cost, because agents ask questions as if they were free, then clips another 0.14 to 0.22 off the final score.

The weaker models leave visible damage. Claude Haiku 4.5 scores below the do-nothing baseline on 79 of 95 tasks. Mistral Small 4 does so on 51 even before stakeholder cost is deducted, and on 23 tasks the delivered system 'caused more harm than the benefit produced by the reference solution.' Extended thinking does not rescue Haiku 4.5: -0.33 without it, -0.35 with it.

FDE-Gym arrives alongside a run of enterprise-realism efforts in our feed this week, including Meituan Longcat's Data World framework for scaling LLM training environments. The paper's own closing line is blunter: 'the last mile of AI delivery remains long.'