Paper: Orthrus 'Lossless' Decoding Only Holds Under FP32

The 'lossless' claim of Orthrus, a hybrid autoregressive-diffusion speculative decoder, holds only under FP32 arithmetic. Under BF16, an independent reproduction posted to Hugging Face finds it reproduces the Qwen3-1.7B reference trajectory in just 45% of prompts for the authors' released checkpoint and 43% for a re-trained variant, across 1,190 prompts drawn from 12 text domains.

Repeating the same evaluation under FP32 flips the result: exact matching on every one of the 1,190 prompts. The authors write, "Under BF16 inference, exact trajectory matching occurs in only 45% of cases for the authors' checkpoint and 43% for our independently trained model across 1,190 prompts from 12 domains." They attribute the BF16 divergence to "finite-precision numerical effects, without attributing them to any particular layer or computational operation."

The drift is not uniform across inputs. A logistic regression yields a coefficient of β1=-8.10 for the released chiennv/Orthrus-Qwen3-1.7B checkpoint and β1=-10.92 for the retrained Orthrus-1.7B-final, both with 95% confidence intervals excluding zero after controlling for response length and prompt domain. Prompts the reference Qwen3 assigns higher response-conditional perplexity are substantially less likely to reproduce exactly.

The drift does not show up in lm-eval-harness. On GSM8K, HumanEval, and IFEval, point estimates for the retrained Orthrus were slightly higher than the Qwen3-1.7B baseline, though the authors caution these should not be read as significant. Their point is subtler: "two systems may obtain identical or statistically indistinguishable task scores while producing different token sequences."

The paper does not withdraw Orthrus as an acceleration technique. The consensus mechanism, the authors write, "can provide exact trajectory equivalence under an idealized arithmetic model, but an implementation using finite-precision arithmetic need not reproduce the reference computation bit-for-bit." Any lossless claim, they argue, should specify the precision and operational criterion under which it was measured.