Nvidia's Dory1.0-4B uses looped layers to match 9B+ rivals
TL;DR
- A single 4B checkpoint supports multiple inference compute budgets by varying how many times a shared block of layers is re-applied.
- With two or more loops, the model's average score across 12 benchmarks outpaces Qwen3.5-9B, Gemma 4 12B, and Gemma 4 26B.
- KV-cache footprint stays equivalent to a non-looped model at any loop count, and four loops cut tokens generated by 1.3-2.5x versus one loop.
Dory1.0-4B is a 4-billion-parameter language model that shares one block of layers and re-applies it a variable number of times at inference, so the same weights can serve different compute budgets. In a paper on arXiv, authors Maxim Fishman, Simeng Sun, Boris Ginsburg and colleagues report that "with two or more loops, Dory1.0-4B outperforms larger models such as Qwen3.5-9B, Gemma 4 12B, and Gemma 4 26B" on an average of 12 benchmarks. Per-benchmark numbers are not in the abstract.
The backbone is a normalized Transformer with interleaved sliding-window and full-attention layers, split into an input block, a shared recurrent block, and an output block. Training ran on 6 trillion tokens, with context extended to 256K, followed by supervised fine-tuning and reinforcement learning with verifiable rewards. BF16, MXFP8, NVFP4 and GGUF checkpoints have been open-sourced.
The paper claims that with four loops the model generates "1.3-2.5x fewer tokens than with one loop," while the KV-cache footprint stays equivalent to a non-looped model whatever the loop count.
Originally reported by paper
Read the original article →Original headline: Nvidia Dory1.0-4B: One 4B Checkpoint, Elastic Inference Loops, Beats 9B+ Models on 12 Benchmarks