Swiftlet Runs 4-bit Qwen3-Next 80B in 4.3 GB RAM on Mac

A single-developer Swift and Metal runtime called Swiftlet is claiming something that would have sounded like a joke a year ago: a 4-bit Qwen3-Next-80B mixture-of-experts model running on a Mac in about 4.3 GB of peak RAM, and a 35B sibling running on an iPhone 17. The project's GitHub reports the 80B decoding at 4.5 to 5 tokens per second, and the 35B at 7 to 11 tokens per second on an M5 Mac. On the phone, the 35B lives inside roughly 2.6 GB of RAM at about 1 token per second.

The trick is not compression, it is streaming. Qwen's routed design only activates about 3 billion parameters per token, so Swiftlet keeps the small dense core resident (around 1.3 GB for the 35B and 2.5 GB for the 80B at 4-bit) and pulls expert weights off disk on demand, cached by a bounded pool that mixes LFU with recency. Experts are repacked into fixed-stride `.qpack` containers so the read pattern plays nicely with an SSD. Attention runs through a gated DeltaNet linear-attention path on 75% of layers, which keeps memory pressure flat as context grows.

Why this matters if you are not writing kernels: the assumed floor for running 80B-class open models has been a workstation GPU or a Mac with tens of gigabytes of unified memory. If a routed-MoE model can be streamed this cleanly, the floor drops to any modern Mac and, for smaller sizes, to a phone. That reshapes what a private, offline assistant can plausibly look like on Apple hardware, and the author already ships an iOS build under the name Priv AI plus an OpenAI-compatible local server and a SwiftletCore library for integration.

The honest caveat is that these are the author's own numbers, not third-party benchmarks, and roughly 1 token per second on an iPhone is a technical demo more than a usable chat UX. The README does not report a quality delta against a non-streamed baseline, or what continuous expert paging does to SSD wear over long sessions. Take the specifics as reported, not settled. The direction, though, is the part worth watching: MoE plus weight streaming is quietly turning the memory budget into a caching problem instead of a hard cap.