Kantamneni, Tegmark: LLMs add by rotating a number 'helix'

Numbers live inside large language models as a 'generalized helix,' according to a new arXiv paper by Subhash Kantamneni and Max Tegmark that reverse-engineers how three mid-sized LLMs do addition.

The authors call the mechanism the 'Clock' algorithm. They write that 'to solve a+b, the helices for a and b are manipulated to produce the a+b answer helix which is then read out to model logits.' They verify the account by modeling MLP outputs, attention head outputs and 'even individual neuron preactivations' with these helices, then running causal interventions.

The helical representation does not stop at addition. The paper reports it 'is also causally relevant for integer division, multiplication, and modular arithmetic.' Kantamneni and Tegmark position the work as 'the first representation-level explanation of an LLM's mathematical capability.'

The abstract does not name the three LLMs or publish per-model accuracy numbers.