AMD Ships 14 Zebra-HyLo Checkpoints, Cuts KV-Cache 92-98%

AMD released 14 open-source hybrid language models on Monday, all built by carving up existing Llama and Qwen checkpoints instead of pretraining from scratch. The technique, called Zebra-HyLo, swaps most attention layers for either Multi-head Latent Attention or a linear block (Gated DeltaNet or Mamba-2), then distills the hybrid back toward its Transformer teacher on roughly 10B tokens of post-training.

The AMD team frames the motivation bluntly in their ROCm blog post: hybrid models 'have become the default answer to the cost of long context' but 'they are all pretrained from scratch. Every one of them pays the full cost of building a foundation model again, which means the enormous investment already sunk into existing Transformer checkpoints is thrown away.'

The KV-cache savings are the headline. Every released checkpoint 'frees at least 92%' of the baseline cache, and the tables post specific reductions of 92.2%, 95.3%, 96.1% and 98.0% across the Llama-3.2-1B, Llama-3.2-3B and Qwen3-1.7B variants. On the RULER long-context benchmark at 64K, the strongest 3B variant, HyLo-Llama-14MLA14GDN, holds 52.0 'with under 5% of the backbone's KV cache.' The 1B and Qwen3-1.7B variants score 40.8 and 33.1 respectively at the same length. On serving, the authors say the models 'complete the sweep to 2M, a 30x context extension over the baseline' on a single node of eight AMD Instinct MI300X GPUs at tensor-parallel 8.

The finding the authors underline isn't the compression itself. It's that context length has to be baked into the recipe from day one. 'A model upcycled at 8K context scores 55.1 on RULER at 8K and 0.8 at 64K — it has essentially no usable long-context behaviour,' they write, before summarising the lesson as 'long context is a training objective, not an architectural gift.'

Code, configs and weights ship under Apache-2.0, with the blog itself noting a caveat on long-context data licensing. It adds to a busy stretch we've been tracking in our AI infrastructure feed and the open-source AI feed.