tomshardware.com via Reddit

Kimi K2.5 1T-param model runs locally on Optane RAM

open source inference local-llm inference hardware

Key insights

  • Intel Optane DIMMs enabled 768GB of addressable memory at low cost, bypassing the need for GPU VRAM entirely.
  • Kimi K2.5's mixture-of-experts architecture reduces active parameters per token, making memory capacity the primary bottleneck.
  • At 4 tokens per second, the setup is viable for research and evaluation but not production inference workloads.

Why this matters

The memory-bandwidth bottleneck has been the primary argument for why frontier-scale models require GPU clusters, and this run challenges that assumption with off-the-shelf server components, lowering the barrier for independent researchers to probe 1T-parameter model behavior without cloud access. For founders building on open-weight models, it signals that the cost floor for capable inference hardware will keep dropping in ways that outpace current pricing assumptions for API alternatives. For technical leaders evaluating open-weight vs. proprietary model risk, it confirms that once weights are released, containment of who can run them at scale is effectively impossible.

Summary

A hardware enthusiast has successfully run Moonshot AI's full 1-trillion-parameter Kimi K2.5 model on a single consumer-class machine using 768GB of Intel Optane DIMMs, achieving roughly 4 tokens per second without any dedicated GPU. Optane DIMMs sit in standard memory slots but offer far higher capacity and lower cost per gigabyte than conventional DRAM, making them an unconventional but surprisingly workable solution to the memory-bandwidth wall that normally forces frontier-scale inference onto expensive GPU clusters. The machine bypasses the need for HBM-equipped accelerators entirely by loading the full model into persistent memory addressable as system RAM. Essentially: (Moonshot AI, Intel) inadvertently enabled a $0-GPU path to 1T-parameter inference. - Kimi K2.5 is an open-weight mixture-of-experts model; its sparse activation pattern reduces the compute per token, making memory capacity the binding constraint rather than raw FLOP throughput. - At 4 tok/s, the setup is too slow for production but fast enough for research, fine-tuning evaluation, and capability probing on hardware costing a fraction of an H100 cluster. - The run appears to be the first community-confirmed local deployment of any 1T-parameter open-weight model, documented on r/singularity. The episode makes concrete what open-weight release advocates have argued abstractly: once weights are public, the hardware ceiling for running them falls faster than anyone plans for.

Potential risks and opportunities

Risks

  • Cloud inference providers (Together AI, Fireworks, Replicate) face pricing pressure if commodity Optane-based local inference becomes a reproducible, documented setup within the next 6 months.
  • Export control frameworks targeting GPU exports to restrict frontier AI access are undermined if CPU-addressable persistent memory achieves comparable inference at scale, leaving regulators without an obvious chokepoint.
  • Moonshot AI and other open-weight model labs may face pressure from investors or partners to restrict weight releases if confirmed local 1T-parameter inference normalizes on non-export-controlled hardware.

Opportunities

  • Intel's Optane DIMM inventory, previously written off as a failed product line, gains renewed commercial interest from AI labs and hobbyists seeking high-capacity memory for large model inference.
  • Memory-optimized inference software vendors (llama.cpp maintainers, Hugging Face, Unsloth) can build Optane-specific quantization and batching optimizations to push throughput beyond 4 tok/s on existing hardware.
  • Independent AI safety and capability researchers gain a credible path to running frontier-scale open-weight models locally, expanding the pool of actors who can conduct alignment and red-teaming work without API rate limits or logging.

What we don't know yet

  • Total hardware cost of the Optane DIMM configuration has not been disclosed, making cost-per-token comparisons to GPU cluster alternatives impossible.
  • Whether Moonshot AI's Kimi K2.5 license permits unrestricted local inference at this scale, or whether commercial use would trigger licensing constraints.
  • Whether the 4 tok/s figure holds across the full context window or degrades significantly as KV-cache grows beyond available Optane bandwidth.

Originally reported by tomshardware.com

Read the original article →

Original headline: Enthusiast Runs Full 1-Trillion-Parameter Kimi K2.5 Locally on 768GB of Cheap Intel Optane DIMMs — Achieves ~4 tok/s Without a GPU