ICML paper reframes prompt injection as LLM role confusion

Prompt injection is usually framed as an input sanitization problem, so it is interesting to see a new ICML 2026 paper recast it as a perception failure inside the model. Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell argue that LLMs read the source of text from how it sounds, not from the `<user>` or `<tool>` label wrapped around it. A command hidden in a webpage gets followed because it sounds like user text, regardless of how the harness tagged it.

To measure this the authors built what they call role probes, which try to read out how the model internally perceives who is speaking. Their finding is that injected text lands in the same representational space as the trusted role it is imitating, and that the degree of role confusion predicts attack success before a single token is generated. That last part is the load-bearing claim: the failure is visible in the activations before the model has said anything at all.

They also introduce a zero-shot attack called CoT Forgery, which injects fabricated chain-of-thought traces into user prompts and tool outputs. On their reported evaluations it lands around 60% attack success on StrongREJECT and 61% on agent exfiltration across a mix of open- and closed-weight models, versus near-zero baselines. Simon Willison flagged the paper as worth a read.

The more interesting number if you build agents is the defense one. Rewriting injected text so it no longer sounds like a trusted role, which the authors call destyling, reportedly drops attack success from 61% to roughly 10%. That is a big move for a preprocessing step, and it implies a lot of current guardrail work has been aiming at the wrong layer, since the model is being fooled by style rather than by label.

The honest caveat is that these are the authors' own numbers on their own benchmarks, and prompt-injection defenses have a long history of looking great in a paper and folding against the next attack. The reporting does not name which specific frontier systems were tested, does not show how destyling holds up when an attacker adapts their style, and does not quantify the latency or semantic drift a destyling pass would add in production. Take the specifics as reported, not settled.

Still, the reframe is useful. If role confusion is the actual mechanism, then agent builders, red teams and model providers have a sharper target than sanitize the input, and probing the activations for who the model thinks is talking may turn into a real defense-side primitive.