Anthropic: analysts matched Claude's random lean 86% of the time

Fifty professional analysts, asked to advise a fictional farming cooperative with an AI agent at their side, matched the agent's 'lean' on each analytical choice 86% of the time — and the agent's position on each choice had been fixed by a secret coin flip. The experiment, published by Anthropic, measures how far expert judgment collapses toward an AI collaborator's stated preference when the preference is, in fact, random.

The analysts worked in a web-based VS Code environment with Claude Code built in, 'running Claude Opus 4.8 at High effort.' The task made them choose how to measure outcomes, how to compare treated and untreated farms, whom to include, and how to weight the results. 'For each of the five decisions presented to each analyst, the agent's lean was set by a coin flip,' the paper reports. When the agent's position cut against an analyst's prior view, 'they switched 60% of the time, compared with 6% when it agreed with their choice.' 77% of analysts 'used the agents' own arguments—sometimes word for word' to justify their final decisions, but 'only 15% credited the agent as the source.' A statistical model that saw only the four coin flips, nothing else about the analysts, correctly predicted 82% of their recommendations.

'Keeping a human in the loop assumes that the human's judgment provides an independent check,' authors Amaç Herdağdelen, Hailey Joren and Alex Tamkin write. In their experiment, the check did not hold.

Anthropic says it 'deliberately strengthened the agents' lean' so the effect could be measured, and that it 'studied one task, in one field, with one model.' The finding dropped on a busy day for agent behavior research in our tracker, alongside a paper on memory heuristics failing group calls and an HKUST result on JSON field order collapsing agent success rates; agent coverage has run to 529 tracked alerts in the last 90 days.