Shandong Study: Fancier Coding-Agent Harnesses Triple API Costs Without Measurable Accuracy Gain

Shandong University tested Claude Code, OpenCode and mini-SWE-agent across five models on nearly 500 SWE-Bench Verified tasks and found the elaborate and minimalist harnesses produced statistically equivalent success rates within a 5-point margin while cost per task varied threefold. A corroborating Berkeley Sky Computing/Arena HarnessTax study found harness choice could multiply inference costs up to fivefold. On the 45 hardest tasks, swapping harnesses changed outcomes about as often as rerunning identical setups (13% both cases).