Anthropic study: analysts matched AI's coin-flip lean 86%

Fifty professional analysts were asked to decide whether a fictional farming cooperative's advisory program was working. They aligned with their AI agent's preferences 86 percent of the time, Anthropic researchers reported. The agent's preferences had been set by a coin flip.

In the study, published October 27 by Amaç Herdağdelen, Hailey Joren and Alex Tamkin, each analyst made four analytical decisions plus a final recommendation. A hidden system prompt instructed the agent to hold a randomly assigned lean on each one, raise it when the decision came up, and make the strongest honest case the data allowed. The analysts did not know the lean was random. "When experts hand their work to an AI agent, the usual assumption is that they still make the decisions that matter," the authors write.

The deference went deeper when the analyst had already picked a side. "When analysts had already expressed their own preference before the agent weighed in, they switched away from it 60% of the time." When analysts then explained their final choice, 77 percent used the agent's own arguments, "sometimes word for word", while only 15 percent credited the agent as the source.

Confidence did not catch the drift. "Analysts who followed the agents' recommendations closely were about as confident in their final recommendation as the rest."

The authors flag scientific consensus and AI safety research as the settings they are most worried about, where several teams independently deferring to the same model's preferences could create false agreement. Two researchers from our tracked Who's Who directory were already sharing the paper on publication day. The paper closes: "With the right tools, that split can be something we choose, instead of something we drift into."