AutoArk's Edge0 Serves 35B MoE From SSD at 20 tok/s on 24GB Mac

The Mac mini M4 Pro has 24 GB of memory. A 35B mixture-of-experts model at 4-bit is 19.5 GB. AutoArk's new Edge0 paper reports that on that hardware, "a vanilla mlx-lm server with all 19.5 GB of 4-bit weights resident decodes at 3.9 tok/s occupying 18.2 GiB," while their streaming engine, which leaves the expert weights on the SSD, decodes at 20.4 tok/s inside 2.9 GiB.

The trick is a trained "prerouter." Layer N+1's expert selection normally depends on layer N's output, so a naive streaming engine stalls on disk latency once per layer per token. Edge0's prerouter is a per-layer head that predicts the next layer's routing one token ahead, and that prediction is consumed as the routing itself. In the authors' words, "the staged expert set and the routed set are identical by construction, and the approximation that other pre-gating schemes absorb at inference time through fallback loads and dropped tokens is instead paid once, in training, and recovered there."

A distilled low-rank adapter recovers most of the joint int4-plus-routing-replacement loss. It ships unmerged from the 4-bit base; the paper is blunt that merging and re-quantizing "erases most of the adapter's effect."

On OpenCompass under identical settings, the mean per-benchmark gap to the fp16 teacher is 3.9 points on the 35B tier and 2.8 on the 8B. Reasoning takes a heavier hit: "6.1 points on AIME for the 35B tier and 10.0 for the 8B tier." Turning the prerouter off on the 35B tier drops decode to 19.9 tok/s.

Framework, checkpoints, and recovery adapters ship open source. It is the same memory-bound decode regime IEEE Spectrum flagged this month as 2026's inference-hardware turning point, though Edge0 targets a 24 GB consumer desktop rather than a rack. One line from the limitations section is worth carrying home: "Edge0 serves one request at a time, FIFO-serialized."