Huginn Sandwich Loop Tops Ouro on 10 Benchmarks at 730M Scale

At the 730M-parameter and 336B-token scale, a new ablation study reports, a Huginn-style sandwich architecture outperforms Ouro on every one of ten benchmarks. The paper from the Institute of Foundation Models, USC and CMU, posted on Hugging Face, isolates the three axes that previous "complete recipe" comparisons left tangled: where recurrence sits in the stack, whether and how the input is reinjected each iteration, and how the recurrent state is organized.

Ouro ties all 28 Transformer blocks and runs the stack four times. Huginn instead places a tied 12-block recurrent core between an 8-layer untied prelude and an 8-layer untied coda, iterated eight times for the same 112 logical block executions. Trained from scratch on the TxT360 dataset at matched parameter count, logical depth and token budget, the paper writes: "at the 730M-parameter and 336B-token scale, the complete Huginn architecture outperforms Ouro across all ten benchmarks."

The advantage carries into Mixture-of-Experts. At the 8BA0.8B scale (32BA3B when unrolled) trained on 500B tokens, Huginn beats Ouro by 4.70 points on GSM8K, 3.60 on MATH500, 2.78 on MMLU, and 1.95 on DROP, while trailing on ARC-Challenge by 0.86. "The Huginn-style MoE outperforms Ouro on eight of ten benchmarks," the authors report, and its load-balancing loss stays below both comparators throughout training.

Against a 112-layer feedforward MoE reference with 32.0B resident parameters, Huginn uses 75% fewer resident parameters and still wins on DROP and GSM8K, matches on MATH500, and remains competitive elsewhere. The mean benchmark gap to the feedforward reference drops from 4.96 points in the dense setting to 1.71 points under MoE. Code is published at github.com/ifm-ai/xllm-loop.

The experiments live at one practical scale. The paper flags what is missing: "a small-to-large scaling ladder should repeat the controlled comparisons across resident parameter counts and training budgets," and "multi-seed replications should quantify the seed-to-seed uncertainty of the reported differences." One ablation also found that prelude-state injection reduces MATH500 by 3.60 points and GSM8K by 2.51 points in the dense setting, where the sandwich envelope compensates. The paper joins a dense run of training-architecture studies we've tracked, 356 of them in open-source over the last 90 days.