FOAM halves SOAP's state while matching Muon wall-clock speed
TL;DR
- FOAM matches SOAP's validation loss on 124M-760M LLM pretraining while using half of SOAP's optimizer state.
- The variant drops the output-side Kronecker factor to remove one eigendecomposition and swaps dense rotated second moments for a row-by-column factored estimator.
- Authors measure a 15-20% K-FAC violation with negligible update effect and robustness across a three-fold learning-rate range.
A NeurIPS 2026 paper by Jaemyung Yu, Byeongho Heo, Sangdoo Yun, and Dongyoon Han proposes FOAM, a lightweight variant of the SOAP second-order optimizer that "uses half of SOAP's optimizer state" and "runs at Muon-class wall-clock time" while matching SOAP and Muon validation loss on LLM pretraining from 124M to 760M parameters.
Two changes carry the cost reduction. FOAM "employs input-only preconditioning, dropping the output-side Kronecker factor to reduce memory and remove one eigendecomposition." The authors argue this is "exact for a diagonal output-side curvature and nearly lossless for a near-diagonal one, as we observe empirically." FOAM also replaces SOAP's dense rotated second moment with "a factored Adam-moment estimator based on row-by-column statistics," which under a K-FAC Fisher assumption with diagonal output curvature "exactly recovers SOAP's dense second moment."
In practice the paper reports "a 15-20% K-FAC violation, with negligible effect on the update," and retains robustness "across a three-fold learning-rate range." The tested scales stop at 760M parameters; the abstract reports no results above that.
Originally reported by paper
Read the original article →Original headline: FOAM: NeurIPS 2026 Optimizer Halves SOAP Memory, Runs at Muon Speed Across 124M–760M LLMs