Cua Metal shim lifts macOS VM llama.cpp to near bare-metal

Running llama.cpp inside a macOS VM on Apple Silicon has always been a compromise. The paravirtualized GPU that Apple's Virtualization.framework hands the guest reports a conservative Metal capability profile, so llama.cpp picks slower kernels and leaves most of the M-series silicon idle. On GitHub, Cua engineers Francesco Bonacci and Johnny Franks describe a small, process-scoped shim that changes those answers for a single guest process and, on their bench, closes almost the entire gap to running on the host directly.

The mechanism is deliberately narrow. The shim intercepts Metal capability queries and reports Apple family 9 instead of Apple 5, and 64 KB of threadgroup memory instead of 32 KB. That is enough to unlock SIMD-group matrix operations, SIMD-group reduction, and bfloat16 support inside the guest, which is what llama.cpp needs to switch to the faster kernels. No physical GPU passthrough is involved.

The numbers are large. On an M1 Ultra with a 48-core GPU, TinyLlama 1.1B prompt processing goes from 431.86 tokens per second in the stock guest to 4,786.70 in the unlocked guest, an 11.08x jump that lands at 98.25% of the 4,871.99 tok/s bare-metal baseline. Token generation moves 16.36x, from 12.63 to 206.60 tok/s. On Google's Gemma 4 12B QAT Q4_0, prompt processing hits 7.20x and token generation 14.54x, with prompt throughput at 99.59% of bare-metal. MLX-LM, tested as a control, saw negligible gains, which the authors read as confirmation that the shim's scope is correctly narrow.

The caveats sit in the write-up itself. The shim relies on private Metal internals that, in the authors' own words, "can change in any macOS release," the validation covers a single M1 Ultra host and Tahoe guest with two llama.cpp models plus one MLX-LM run, and hardened executables may refuse the library injection outright. Cua has invited Apple engineers to email [email protected], which reads like the authors would rather see the underlying capability probe fixed upstream than keep patching around it.

For teams building local-inference dev loops or agent-testing CI on Apple Silicon fleets, this is a way to keep VM isolation without paying an order-of-magnitude tax on GPU work, and it lands in the middle of a busy stretch of Apple silicon coverage this month, our 89th such story in the last 90 days. The bigger question is whether Apple treats the post as a bug report or as unauthorized poking at private surfaces.