Orthrus 'lossless' decoding holds only under FP32, paper finds

Orthrus, the hybrid autoregressive-diffusion decoder whose central claim is lossless speculative decoding, produces the same output as vanilla autoregressive inference in only 45% of cases when run at BF16, according to an independent replication by Ilya Koziev, Leonid Sinev and Ivan Oseledets posted to arXiv.

The authors reproduced Orthrus and ran 1,190 prompts across 12 domains against both the original authors' checkpoint and their own independently trained model. Exact trajectory matching held in 45% of cases for the released checkpoint and 43% for the reimplementation. Repeating the same evaluation in FP32 gave, in the paper's words, "exact trajectory matching on all evaluated prompts."

The divergence did not show up on standard task benchmarks. "Despite this trajectory divergence, Orthrus does not show systematic degradation on downstream lm-eval-harness benchmarks," the authors write. Where trajectories did diverge, the probability of an exact match was "strongly associated with the response-conditional perplexity of the reference model," meaning outputs drifted most on prompts the reference model was already least sure about.

The conclusion the paper draws is narrow but pointed: "the practical losslessness of Orthrus depends on numerical precision," and "exact trajectory equivalence should be evaluated separately from downstream task performance."