NYU Paper Pushes Back on LLM Introspection Claims

A short paper out of NYU's Center for Data Science, posted to arXiv by Shashwat Singh, Tal Linzen and Shauli Ravfogel, is a direct pushback on a run of recent results that suggested large language models can introspect on their own internal states. The authors' bottom line is that the current evidence is insufficient to establish strong metacognitive monitoring, and that when you tighten the controls the effect largely goes away.

They go after two strands of work. The first is the biofeedback paradigm, where a model is asked to predict labels that were derived from its own hidden state activations. The paper's argument is that those labels are 'largely predictable from input features' too, so a model succeeding on the task may just be doing in-context learning of the semantic task rather than monitoring itself. To test that, they retrain with random-relabeled versions that strip the semantic correlations, and report that performance falls close to the majority-class baseline. They also note that a linear probe on layer-0 embeddings can match or exceed the model's own accuracy on the Belief Dominance metric they examine.

The second target is the steering-detection paradigm associated with Lindsey (2025), where the model is supposed to notice when an activation-steering vector has been injected into its internals. Singh and colleagues add what they call a 'gaslight' condition, an input-level manipulation designed to look weird without touching activations, and report that the model 'fails to reliably distinguish input-level from activation-level interventions.' Their reading is that models are picking up on general anomaly rather than genuine self-monitoring.

Why this matters for anyone building on frontier models: introspection is doing load-bearing work in interpretability and safety pitches, from confidence estimates to self-reports of intent. If the underlying signal is really 'this prompt or activation feels off,' the reliability story behind those product claims is much thinner than it reads. The honest caveat is that this is one methodological paper on arXiv rather than a settled community verdict, and the authors do not argue that no form of introspection is possible, only that the current benchmarks do not prove it. What the paper does not give you is a positive test that would count, and defining that test is the interesting next fight.