llama.cpp merges FP8 KV-cache support for OCP-FP8 models
TL;DR
- ORippler's PR #28392 adds OCP-FP8 KV-cache support to llama.cpp, with optional runtime, static per-tensor scales.
- The change uses graph-level scaling instead of touching `ggml_flash_attn_ext`, keeping the ownership model consistent with the existing NVFP4 and FP8 logic.
- Tests cover conversion, loading, context shifts and FP8-scaled correctness; NVIDIA's Mistral-Medium-3.5-128B is cited as an example with non-unit-scaled FP8-KV-cache.
PR #28392, merged into llama.cpp by contributor ORippler, adds support for OCP-FP8-based KV-caches with optional runtime, static per-tensor scales. The implementation uses graph-level scaling rather than modifying `ggml_flash_attn_ext`, which the author says "matches the ownership-model we have in place for NVFP4/FP8, and makes it easier to share FP8 logic across FA and standard attention layers."
Tests cover conversion and model loading, context shifts, and general correctness of FP8-scaled KV caches, plus dequantizing FP8 before the rope shift. Reviewers ggerganov and am17an approved after minor adjustments including renaming cache scale references and allowing BF16 scales alongside FP8. The PR cites NVIDIA's Mistral-Medium-3.5-128B as an example model with non-unit-scaled FP8-KV-cache.
It lands on the same project that shipped v0.9.0 earlier this month with an OCP FP8 E4M3 tensor type, slotting into a short run of local-inference memory work on our tracker alongside Apple's Harbor Thunderbolt-5 Mac clustering a few days earlier.
Originally reported by github.com
Read the original article →Original headline: llama.cpp Merges FP8 KV-Cache Support, Trimming Memory Needs for Long-Context Local Inference