MBZUAI Opens K2-Horizon Fleet, Six Models From 0.9B to 375B
TL;DR
- MBZUAI released six fully open K2-Horizon models from 0.9B dense to a 375B-A23B MoE, with intermediate checkpoints, pretraining data and pipelines under LLM360.
- The 375B-A23B scores 65.3 on Toolathlon-Verified and 48.4 on SWE-Atlas-QnA, surpassing GPT-5.6 Terra on the former while trailing closed systems on scientific-reasoning tests HLE and CritPt.
- A new Mixture-of-Value Attention design lets the 36B-A4B MoE nearly match a 32B dense model on the same corpus with one-seventh the active parameters.
MBZUAI has released K2-Horizon, a fleet of six fully open language models spanning a 0.9B dense network through a 375B-A23B mixture-of-experts flagship, with every intermediate training checkpoint, roughly 10T synthetic pretraining tokens, and the data pipelines that produced them bundled into the release. The paper on Hugging Face frames the drop against the LLM360 standard for open source, which it says covers "the full development history, including intermediate checkpoints from every training stage, pre-training data, training logs, and all training, evaluation, and infrastructure code."
The headline benchmark is the 375B-A23B's score of 65.3 on Toolathlon-Verified, which the paper reports leads GLM-5.2, MiniMax M3, Nemotron 3 Ultra and Inkling, and surpasses GPT-5.6 Terra. It also posts 48.4 on SWE-Atlas-QnA on repository-level code. The authors concede the main gap against closed competitors sits on expert-level scientific-reasoning tests HLE and CritPt.
On the modeling side, the authors write, they "introduce Mixture-of-Value Attention (MoVA), applying MoE to attention value projections, and softplus attention gating." Their 36B-A4B model, which uses both, "nearly matches a 32B dense model trained on the same data, with one-seventh the active parameters." The team also ships Uno, "lossless diffusion adapters that speed up decoding by up to 2.2× without a separate draft model," for the 0.9B and 7B tiers.
Four of the six models (3.7B, 7B, 32B and 36B-A4B) share a single 22T-token pretraining corpus and recipe, which the paper argues enables "controlled studies of scale and sparsity" that are usually impossible on open weights. The 375B-A23B separately trains on 15.1T tokens across two phases on NVIDIA H200 GPUs, and ships FP8 and NVFP4 quantizations; smaller models release in GGUF and MLX for local inference.
The drop lands in the middle of a heavy run of open-model releases on our open-source tracker, 383 stories in the past ninety days, but K2-Horizon is unusual in attaching the pipelines that produced the data, which the paper calls "the part of model development that open releases most rarely disclose." All artifacts are published at ifm.ai/k2/, with correspondence to Hector Liu and Eric Xing.
Originally reported by huggingface.co
Read the original article →Original headline: MBZUAI Opens K2-Horizon Fleet: Six LLMs Up to 375B With Full Checkpoints, Data and RL Stack