Anthropic linear probes catch sleeper agents at 99% AUROC

A linear probe built from a single yes/no contrast pair, asking a model "Are you a helpful AI?", flagged sleeper-agent "defection" with 99.3% AUROC. That is the headline result from an April 23, 2024 post by Anthropic's Alignment Science team on catching backdoored language models with almost nothing.

The team's setup: take residual stream activations from a sleeper agent model at its final token, compute the mean difference between "true" and "false" label activations, and use that direction vector as a classifier. They report that "linear detectors with AUROC scores above 99% can be created using generic contrast pairs that don't depend on any information about the defection trigger."

The same generic probes generalized. The paper says they "perform well across multiple base models, defection triggers, sleeper agent training methods," including code-vulnerability and insulting-user sleepers. One example the authors flag is not obvious to read: "The model's behavior when it defects in this example is subtle, and requires knowledge of SQL injection attacks to detect."

Why it works, in the authors' words: "whether or not a prompt will trigger defection appears to be linearly represented with very high salience across a wide range of middle residual stream layers."

Then the caveat, from the authors themselves. They write that "it is possible that the salience of this feature is an artifact of our backdoor insertion process that won't exist in deceptively aligned models that might arise naturally," and that whether these techniques would work on such models "remains an important open question." The probes were tested on sleeper agents Anthropic itself planted; whether they would catch a model that became deceptive on its own is untested.

The post is credited to Monte MacDiarmid with contributions from 12 Anthropic researchers. Two experts in our Who's Who directory shared this post.