'Thinking Inertia': LLMs keep reasoning under no-think controls

Explicit 'no-think' controls on large language models do not reliably stop visible inference, and the failure grows worse as the answer space opens up, according to a new arXiv preprint titled 'Thinking Inertia: LLMs Keep Thinking When Told Not To.' The authors evaluate six prompting interventions on six LLMs across Boolean, multiple-choice, and open-ended questions, and argue that proxies used in prior work, such as a disabled thinking mode or the absence of long traces, are unreliable because 'disabled thinking modes may still emit reasoning, while long traces may contain filler rather than genuine inference.'

In place of those proxies, Lei, Lin, Lu, Torr and Zou propose three metrics: an Empty-Thinking Rate for strict answer-only compliance, an instruction-aware Question-Pre-answer Relevance score, and an LLM-as-judge Explicit Inference Rate. Across the six models, 'explicit no-think controls cannot reliably eliminate visible inference,' and the paper names the pattern 'Thinking Inertia': explicit inference persists even under strict controls and becomes more prevalent as the answer space opens.

The accuracy picture splits by task shape. On Boolean and multiple-choice questions, accuracy stays stable even when models comply with answer-only instructions. On open-ended questions, the paper reports 'a trade-off between answer-only compliance and task accuracy.' Rewriting the same question into a format with candidate answers, the authors add, 'makes answer-only responses easier to produce.'