'Looped Models Done Right' Part I: Huginn-Style Sandwich Beats Ouro at Matched 730M Scale
Summary
Part I of 'Towards Looped Models Done Right' isolates recurrence placement, input injection and recurrent-state organization in looped language models at matched parameter scale. The prelude-recurrent-coda 'sandwich' architecture (Huginn-style) outperforms Ouro at 730M/336B-token scale across all 10 benchmarks, with gains transferring to 8BA0.8B MoE on 500B tokens.
Originally reported by huggingface.co
Read the original article →Original headline: 'Looped Models Done Right' Part I: Huginn-Style Sandwich Beats Ouro at Matched 730M Scale