Li, Webb: data, not architecture, drives LLM symbolic reasoning
TL;DR
- Training neural nets from scratch on abstract sequence tasks, greater data diversity produced stronger symbolic signatures across multiple axes.
- The same symbolic signatures emerged in modified transformer architectures and even in multilayer perceptrons, not just standard transformers.
- Consistent patterns also appeared in an entity-attribute binding task, supporting the claim that architecture is not the primary driver.
Data diversity, not architecture, is the main lever behind the symbolic mechanisms that let large language models do abstract reasoning. That is the headline claim of a new arxiv preprint from Melody Zixuan Li and Taylor Webb.
The authors trained neural networks from scratch on abstract sequence tasks, varying both the architecture and the shape of the training distribution. They then ran representational, attentional and causal mediation analyses to look for the fingerprints of symbolic computation.
What they report cuts against a common intuition. 'Across multiple axes of data diversity, greater diversity produced stronger symbolic signatures,' the paper says. More striking: 'the same signatures emerged in modified transformer architectures and even multilayer perceptrons, indicating that architectural inductive biases are not necessary for the development of symbolic mechanisms.' Consistent patterns showed up on an entity-attribute binding task too.
The authors land on a flat conclusion: the experiments 'identify data diversity, not architecture, as the primary driver of emergent symbolic computation in neural networks.'
The abstract reports no per-task accuracy numbers, and the models were trained from scratch rather than at frontier scale.
Shared on Bluesky by 1 AI expert
-
🧵How do LLMs develop symbolic reasoning mechanisms? Is it the architecture, or the training data? Excited to share our #NeurIPS2026 paper! 📄 Key takeaway: Data diversity is a primary driver of symbolic mechanisms. Surp…
View on Bluesky →
Originally reported by arxiv.org
Read the original article →Original headline: What drives the emergence of symbolic mechanisms in large language models?