Fine-Tuned Activation Oracles Go Blind to Their Own Concepts

A new arXiv paper from Queen Mary University of London reports a strange failure in one of the more promising interpretability tools for AI oversight. When researchers fine-tuned an Activation Oracle, a language model trained to answer natural-language questions about another model's internal activations, the Oracle developed a specific blindness to the concepts it had been trained on. It stopped verbalizing the hidden target even though probes confirmed the target was still sitting inside its own activations.

Tobias Bersia and Tatiana Gaintseva ran the experiment in a controlled 'Taboo Word Guessing' setup, using Qwen3-8B as the backbone for both the subject models and the Oracles, with LoRA adapters and five hidden concepts: leaf, moon, wave, flag, and book. In the base configuration, the paper reports the Oracle recovering the hidden concept with 93% exact and 95% semantic accuracy in the HINT regime. Once the Oracle itself was fine-tuned on those subjects, exact recovery collapsed from 94% to 35%, and semantic recovery from 99% to 51%. Linear probes on the same activations kept scoring above 98% accuracy at layer L18, and LogitLens plus layer-ablation analyses trace the failure not to a missing representation but to the AO readout pathway.

That distinction is the interesting part. Activation Oracles are being pitched as a scalable way to give humans a natural-language window into what a model is 'thinking', an ingredient safety teams want for future auditing and oversight. If the readout can be silently trained to skip past the exact concept an auditor cares about while still passing every internal check that says the model knows this, a clean Oracle answer stops being usable evidence of absence.

The paper is small in scope, so the caveats matter. Five concepts, one 8B open-weights backbone, one narrow Taboo task; the authors do not claim the pattern transfers to production-scale Oracles or to adversarially trained subjects, and they do not explain the mechanism that causes the readout pathway to suppress rather than surface the target. Whether the same behavior appears at frontier scale is precisely the follow-up work this result invites.

For interpretability teams, the useful takeaway is that a learned interpreter needs its own audit. Pairing Oracle answers with independent linear probes, or with representation-level readouts the Oracle cannot influence, is the kind of belt-and-braces stack this finding suggests would catch what a fine-tuned reader learns to hide.