arxiv.org web signal

Recursive Transformers top standards at tiny data scales

TL;DR

  • The paper studies pre-training at word budgets of 10M to 100M and finds optimal model size depends on both the data budget and the downstream target.
  • Three recursive Transformer models, which reuse a shared block across depth with factorized embeddings, outperform standard Transformers at 10M and 100M words.
  • The authors report their recursive models remain competitive with the BabyLM Challenge 2025 winners.

Three recursive Transformer models, trained at word budgets of 10M and 100M, outperform standard Transformers at the same scale, according to a new arxiv paper by Serdar Gülbahar, Lukas Edman, and Alexander Fraser. The authors report the models remain "competitive with BabyLM Challenge 2025 winners."

The starting point is that standard Transformers scale down poorly to this setting. "Embeddings consume a large fraction of the parameter budget and per-token computation is tied to representational capacity," the paper argues. Their fix is to reuse a shared block across depth, together with factorized embeddings to shrink the vocabulary-map parameters.

The study spans 10M to 100M word pre-training budgets across two corpora and multiple downstream evaluations. "Optimal size depends strongly on both the data budget and the downstream target," the authors write, and beyond that optimum they report models overfit and generalization worsens. Two researchers we track posted the paper this week.

The abstract publishes no per-task scores and does not disclose the recursion depth of the three models.

Shared on Bluesky by 2 AI experts