marktechpost.com via Reddit

Together AI's OSCAR shrinks KV cache memory 8-fold

together ai inference open source kv-cache quantization inference-efficiency

Key insights

  • OSCAR achieves 8x KV cache memory reduction at 2-bit precision while preserving near-BF16 output quality in long-context LLM serving.
  • Per-layer offline covariance analysis distinguishes OSCAR from standard Hadamard rotation by neutralizing channel outliers specific to each layer's activation statistics.
  • Together AI open-sourced OSCAR alongside an arXiv paper targeting high-concurrency deployments where KV cache dominates total GPU memory consumption.

Why this matters

KV cache memory is the binding constraint for GPU utilization in production long-context deployments, and an 8x reduction directly changes the unit economics of serving large models at scale. Operators running high-concurrency workloads can now fit more concurrent sessions per GPU or extend context length within the same memory envelope, shifting competitive dynamics for inference providers who have used memory efficiency as a pricing lever. The open-source release also compresses a previously research-grade technique into deployable infrastructure, setting a new baseline for 2-bit quantization quality at a moment when context windows across frontier models are expanding rapidly.

Summary

Together AI has open-sourced OSCAR, a 2-bit KV cache quantization system that cuts GPU memory 8x while maintaining near-BF16 output quality on long-context workloads. The core problem at extreme bit widths is channel outliers in KV activations, which cause quality collapse. Standard approaches apply a fixed Hadamard rotation uniformly across all layers. OSCAR runs offline covariance analysis per layer and derives custom rotation matrices calibrated to each layer's actual activation statistics, eliminating outliers at their source before quantization occurs. Essentially: (Together AI) made per-layer calibrated rotation practical enough to ship as open-source inference infrastructure. - 8x memory reduction enables more concurrent users or longer context windows on the same GPU budget. - Per-layer offline covariance calibration is what separates OSCAR from generic Hadamard-rotation approaches used in prior quantization work. - KV cache now dominates GPU memory in high-concurrency, long-context serving, making compression a direct operational cost lever. As context windows scale toward millions of tokens, KV cache compression becomes foundational serving infrastructure rather than an optional optimization.

Potential risks and opportunities

Risks

  • Inference providers that built proprietary KV cache compression as a cost moat (Fireworks AI, Groq) face commoditization pressure as OSCAR raises the open-source baseline across the serving stack.
  • Deployers adopting OSCAR without per-application benchmarking could encounter accuracy regressions in safety-critical or precision-sensitive production workloads where near-BF16 is not close enough.
  • Hardware vendors whose revenue depends on memory-dense GPU configurations (SK Hynix, Samsung HBM) face reduced demand pressure as memory efficiency improvements shrink per-deployment VRAM requirements at scale.

Opportunities

  • Inference-as-a-service providers (Together AI, Fireworks AI, Modal) can leverage OSCAR immediately to reduce per-token serving costs and offer longer context windows without additional GPU procurement.
  • Model optimization tooling vendors (Neural Magic, Hugging Face) can integrate OSCAR's per-layer offline calibration into existing quantization pipelines as a differentiated enterprise feature.
  • Enterprises running on-premise LLM deployments gain a concrete path to extending context length within fixed GPU budgets, opening internal procurement conversations for additional model deployments rather than additional hardware.

What we don't know yet

  • Accuracy benchmarks across model families beyond the arXiv evaluation set have not yet been independently reproduced by third parties.
  • Whether OSCAR's offline calibration overhead is tractable for frequently updated or fine-tuned production models remains unaddressed in the initial release.
  • Production throughput and latency numbers at target concurrency levels are absent from both the open-source release and the arXiv paper at 2605.19660.

Originally reported by marktechpost.com

Read the original article →

Original headline: Together AI Open-Sources OSCAR: Attention-Aware 2-Bit KV Cache Quantization With 8× Memory Reduction at Near-BF16 Quality