sFNX runs Qwen3.8-Flash-Next at 382 tok/s on $1,800 V100 rig

"Qwen3.8-Flash-Next at 382 tokens a second on a code edit, for $1,800 of used GPUs." That is the headline claim on sFNX, a CUDA inference engine open-sourced under MIT by a developer going by johnfink8. The rig: three Tesla V100-32GB cards at about $600 each, on PCIe Gen3 with no NVLink.

What is unusual is that every transformer layer is sharded across all three cards rather than each card holding whole layers. The README spells out the split: "a third of each expert's FF width per card" across 512 experts a layer, "16 value heads per card" for recurrent layers, K/V cache "by position: blocks of 4 dealt round-robin, partial results merged," and "a third of the vocabulary per card" at the output head. The author's rationale: "Splitting every layer buys all three cards' bandwidth and costs a sum across them twice per layer, which NCCL cannot pay for over PCIe. So sFNX does the sums itself." The partial sums move between GPUs through custom CUDA-graph writes around a ring rather than a collective library.

Reported benchmarks: 367–382 tok/s for code editing with greedy decoding, 200–229 tok/s for new code generation, 139 tok/s for prose sampling at temperature 1.0, and 280–291 tok/s combined across six concurrent conversations each holding the full 262,144-token context. A 114k-token prompt cold-reads at 3,373 tok/s.

Constraints are concrete. The engine only runs on three sm_70 cards, VRAM is "essentially full at idle," and the README flags that a prompt re-read on a lane "can differ in its last bits (~1e-4 first-token KL)." The release lands into a run of local-inference tooling our tracker has catalogued recently, including Zagitta's gpu-unlock patch that lifts FP16/BF16 caps on RTX 3000-series cards.