IFM, USC, CMU: Latent Feedback Cuts Pretraining Compute 48%
TL;DR
- A new scaling-law paper claims latent feedback saves at most 48% of the compute and 47% of the unique data versus a matched vanilla Transformer.
- Fitted on 60M-271M models, the joint scaling law predicts held-out 778M losses to within 0.3% and the gap between the two architectures to within 0.05%.
- Savings only emerge past about 33 unique tokens per parameter at two epochs, a regime the authors say today's 1B-10B dense models are far past.
A new preprint from the Institute of Foundation Models with coauthors at USC and CMU argues that feeding each token's top-layer state into the next token's input saves, at most, 48% of the compute and 47% of the unique data compared with a vanilla Transformer that spends the same FLOPs on more epochs. Chaitanya Dwivedi, Xuezhe Ma, Alexander Moreno and Hector Liu posted the paper, titled "Latent Feedback Is a Compute and Data Amplifier," on November 11, 2026.
The pitch is that when high-quality text runs out, the common fix of training for more epochs gives diminishing returns. "Each repeat is worth less than the last," the authors write. Their alternative routes each token's final hidden state back into the next token's input through a learned map, costing extra forward passes per update but letting each token do more work.
The savings are a function of how much unique data each parameter sees. At two epochs, latent feedback costs 8% more than the Transformer baseline at 12.5 unique tokens per parameter (UTPP), but saves 15% to 18% at 100 UTPP and 44% to 48% at 400 UTPP. On the data side, at one epoch on 50 UTPP it saves 16% of unique tokens at the 271M size and 20% at 778M; at three epochs on 100 UTPP it saves 41% at 271M.
To go beyond the grid, the authors fit a single scaling law in which latent feedback enters as a Transformer with more parameters. Fitted on models from 60M to 271M, it "predicts the held-out 778M losses to within 0.3% and the gap between the models to within 0.05%." They argue today's practice is well past the crossover: Chinchilla used 20 training tokens per parameter, but the median 1B-10B dense model among 257 base model reports they surveyed trains on 1,040 TPP, at least 260 UTPP even over four epochs.
The code is published at github.com/ifm-ai/LoRM. The abstract reports no per-benchmark downstream numbers, only that "downstream tasks agree," and the ladder tops out at 778M parameters, well below frontier training runs. It lands among a steady run of training-efficiency work on our AI infrastructure tracker.
Originally reported by huggingface.co
Read the original article →Original headline: IFM Paper: Latent Feedback Saves Up to 48% of Pretraining Compute Versus a Vanilla Transformer