Shandong Study: Fancier Coding Harnesses Triple Cost, Not Accuracy

The harness around a coding agent, meaning its system prompt, tool set and context-management rules, can triple what it costs to solve a problem without the agent getting any more accurate. That is the headline from Yangze Liu and Zhongyi Han of Shandong University, whose paper "What Does a Harness Buy? Tokens, Mostly." ran five models through three production harnesses on SWE-bench Verified and then reran the same configurations to see how much a score moves when nothing changes at all.

On the 447-task pool, Claude Code and the minimalist mini-SWE-agent landed within two percentage points of each other and were "statistically equivalent within a five-point margin." On the 45 hardest tasks, swapping the harness flipped which problems the agents solved about as often as rerunning an unchanged setup: 13% of tasks in either case. VentureBeat reports that with the same model, the same tasks, and one price list, "cost per task differs by up to 3x across harnesses."

The driver is fixed context. Each harness ships its system prompt and tool schemas along with every turn, over and over. The paper measures those preambles at "16,581 tokens in Claude Code, 7,025 in OpenCode, and 829 in mini-SWE-agent."

"Every harness effect we could measure is a way to lose a task the model would otherwise solve," the authors write.

A separate paper cited in the writeup, HarnessTax from UC Berkeley's Sky Computing Lab and Arena, evaluated seven models across Claude Code, OpenAI's Codex CLI and Pi, a minimal open-source harness with just four basic tools. It "found that harness choice shifted pass rates by only a few percentage points while multiplying inference costs up to fivefold." The Shandong team also notes that 10 reruns of an identical configuration on SWE-bench Verified produced score spreads of 2.2 to 6.0 percentage points, wider than most of the vendor-advertised gaps leaderboards use to rank them. VentureBeat adds one more number worth sitting with: 21% of surveyed enterprises have no automated way to stop their agents' token spending.