Circuit Grafting Pushes Qwen 3.5 4B to 100% on 6-Digit Math
TL;DR
- Qwen 3.5 4B hits 100% and Granite 4.2 3B hits 99.17% ALU-prefix accuracy on 6-digit signed arithmetic, versus 42.2% to 61.7% for a 12.8K-sample LoRA.
- Grafts modify under 0.06% of weights, pass all 14 pre-registered Tango language splits across 4,392 samples, and nudge Granite GSM8K up 0.99 points.
- Authors flag dual-use risk: bit-exact off-trigger gates and under 0.002 nats KL shift would make dormant payloads undetectable by behavioral benchmarks.
A new preprint compiles exact arithmetic circuits straight into pretrained LLM weights without any training data or runtime tool calls, pushing Alibaba's Qwen 3.5 4B to 100% ALU-prefix accuracy on 6-digit signed arithmetic and IBM's Granite 4.2 3B to 99.17%, versus 42.2% to 61.7% for a 12.8K-sample LoRA baseline. The paper, "A CPU Inside an LLM", posted November 20, describes closed-form, zero-gradient primitives that "compile deterministic computation directly into pretrained LM weights."
The grafts rest on a two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, signed Chebyshev Indefinite-RoPE self-sink attention, and an axis-aligned width extension that keeps per-layer norm drift bounded.
Language competence appears to survive. The authors report 14 of 14 passes on pre-registered Tango retention splits across 4,392 samples, with Granite GSM8K rising 0.99 percentage points and ARC-C 0.09. Active non-zeros come in at 1.07 to 1.33 million for Granite and 2.25 to 2.39 million for Qwen, which the paper puts at 0.036% and 0.057% of weights respectively. A factorized block-sparse sidecar cuts A100 memory from 13.55/19.09 GiB to 7.89/10.65 GiB, a 42% to 44% saving, with 2K-token prefill speedups of 1.96x to 2.67x.
The paper is unusually candid about dual-use risk. It flags "bit-exact zero off-trigger gates" with a next-token KL shift under 0.002 nats as properties that would make dormant input-triggered payloads "undetectable by behavioral benchmarking alone," and proposes white-box structural auditing (dimension verification, L₀ sparsity scanning, SHA-256 provenance) as the mitigation. Not every architecture grafts cleanly: a 42-layer Gemma 4 E4B audit shows 47.9% failure on layers 24 to 41 from a KV-head overwriting bug, and division circuits hit 56.1% under BF16 against 100% with an FP32 sidecar. The authors list full Gemma 4 arithmetic, division, and 9-18 digit scaling as open work.
Originally reported by huggingface.co
Read the original article →Original headline: Zero-Gradient Compiler Grafts Exact Multi-Digit Arithmetic Circuits Into Pretrained LLM Weights