ANEMLL pairs M6 Neural Engine and GPU for faster Qwen prefill
TL;DR
- ANEMLL's hybrid pipeline delivers 24 to 40% faster prefill than a GPU-only baseline on an M6 Mac across context lengths from 2K to 96K tokens.
- Whole-machine energy above idle drops 41 to 50% per generated token and 15 to 24% per prompt token at 8K to 96K, with decode draw of 20 to 24 W versus 38 to 40 W.
- The build only runs with System Integrity Protection turned off so it can read the Neural Engine's compiled weight cache, limiting it to dedicated research machines.
ANEMLL researchers running a 27-billion-parameter Qwen3.8 model on a single 32 GB Mac with an M6 chip report that splitting each transformer layer between the Neural Engine and the GPU, with feed-forward networks on one side and attention mixers on the other, pipelined over 64-token blocks, makes prefill 24 to 40 percent faster than a GPU-only baseline at every context length they tested from 2K to 96K tokens.
The post frames the architecture plainly: "FFNs on the Neural Engine, mixers on the GPU, both busy at once." The headline figure is "685 against 489 tokens per second at 8K and 389 against 314 at 96K," with decode "within -4% to +6% of Splash at every length." Energy tells the louder story: "41 to 50% less whole-machine energy above idle per generated token, and 15 to 24% less per prompt token at 8K to 96K." Decode power draw above idle falls from 38 to 40 W on the GPU-only baseline to 20 to 24 W under the split.
In a live coding session titled "game of tetris in HTML, test and fix," the setup wrote a 4,613-token single-file Tetris program over 21 tool calls, averaging 27.5 tokens per second end-to-end at context lengths between 6.1K and 27.6K.
The ceiling is memory bandwidth, not compute. "Attention at 60K reads about 93 GB/s from DRAM and the FFN about 85 GB/s. Together they ask for about 180 GB/s from a memory that delivers about 155," the authors write, which is why the best prefill-only run topped out at 745 tokens per second at 2K, "short of the 1,000 we hoped for."
Two caveats sit on top of the numbers. "The Neural Engine slows 30 to 45% after about four minutes of continuous work," making long agent runs a thermal gamble on this hardware. The build also "needs SIP turned off and root permission for weights" to read the Neural Engine's compiled .hwx files out of /Library/Caches/com.apple.aned/, which means it runs only on a dedicated research machine with System Integrity Protection disabled from macOS Recovery. Two of the researchers we track were circulating the post within a day of it going up.
Shared on Bluesky by 2 AI experts
-
Worth paying attention to if you care about local AI a vein of research speeding up performance on M6 huggingface.co/blog/anemll/...
View on Bluesky →
Originally reported by huggingface.co
Read the original article →Original headline: One model, two accelerators: faster prefill and decode on the M6 with the Neural Engine and the GPU together