Olmo Hybrid: arxiv.org/abs/2604.03444 Cracks in foundation: arxiv.org/abs/2608.10296
AI Weekly's analysis
→
- The paper introduces Olmo Hybrid, a 7B-parameter model that keeps the Olmo 3 7B recipe but replaces its sliding window layers with Gated DeltaNet layers.
- Olmo Hybrid outperforms Olmo 3 across standard pretraining and mid-training evaluations, and the authors say it scales more efficiently than the pure transformer.
- Theoretically, the authors argue hybrids can express tasks beyond both pure transformers and linear RNNs, such as code execution.
Read full analysis →
