LLMs Recognize Impossible Questions but Answer Them Anyway

Large language models routinely produce confident answers to prompts that have no correct answer, computing cot(-540°) or resolving (1).startswith("1") on an integer, and a new arxiv paper by Yucheng Du and Xiyang Hu argues the failure is not what most safety pipelines assume it is. The models can tell the questions are impossible; the recognition just never reaches the part of the network that decides to refuse.

Across models ranging from 1.7B to 70B parameters, the authors identify "a single linear direction in the hidden state" that separates answerable from structurally impossible math and code prompts. That direction is present before generation begins. Steering along it "changes invalidity-aware behavior bidirectionally and dose-responsively" on structural tasks, while random directions show no effect.

The catch is where that recognition direction sits geometrically. It operates nearly orthogonally to the safety-refusal direction that harmful-content refusal training shapes. The internal representation for "I cannot answer this" that RLHF has been sculpting is a different vector from the one that actually encodes impossibility.

The misalignment, the paper reports, already existed at the pretraining stage. Later refusal training was never going to route recognition into refusal, because the two live in different subspaces of the hidden state, which is why the authors frame the problem as one of signal routing rather than an encoding failure.