Tikhonov Paper: Transformers Superpose Two Text Streams at Once
TL;DR
- A new arXiv preprint argues that transformer LLMs superpose next-token distributions when two input text streams are linearly combined.
- The authors call it the 'Superposition Linearity Hypothesis' and argue it is intrinsic to the architecture, not learned during pretraining.
- A guided decoding procedure disentangles the superposed output, yielding two coherent continuations from a single forward pass.
Feed a transformer LLM two text streams linearly combined at the input, and the output is a superposition of what each stream would have produced alone. That, according to a preprint posted Thursday to arXiv, reflects a fundamental linearity property of the transformer architecture rather than an artifact of training.
The paper, titled "Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs," calls this the "Superposition Linearity Hypothesis." Its nine-author team, including Pavel Tikhonov, Anton Korznikov, Matvey Mikhalchuk, Ivan Oseledets and Elena Tutubalina, argues the effect diminishes over the course of pretraining but "can be restored through fine-tuning" with lightweight interventions.
The concrete artifact is a decoding trick. The authors introduce "a guided decoding procedure that disentangles superposed outputs, enabling the simultaneous generation of two coherent continuations from a single forward pass."
Notably absent from the abstract: model sizes, benchmark numbers, or any measurement of how the linearity-restoring fine-tune affects standard capabilities.
Originally reported by arxiv.org
Read the original article →Original headline: Paper Argues Transformers Hold Two Thoughts at Once, Shows Linearity Can Be Restored via Light Fine-Tuning