IFM Paper: Latent Feedback Saves Up to 48% of Pretraining Compute Versus a Vanilla Transformer
Summary
Researchers at the Institute of Foundation Models, USC and CMU show latent feedback — piping each token's top-layer state into the next token's input — matches a Transformer's loss with up to 48% less compute and 47% less unique data under compute-matched comparisons. A joint scaling law fitted on 60M-271M models predicts held-out 778M losses to within 0.3% and the gap between the two architectures to within 0.05%.
Originally reported by huggingface.co
Read the original article →Original headline: IFM Paper: Latent Feedback Saves Up to 48% of Pretraining Compute Versus a Vanilla Transformer