Greek MoE reasoning study: seed variance beats training effects

On accuracy benchmarks, fine-tuning three frontier mixture-of-experts models to reason in Greek produced almost no measurable movement, and the authors argue the metric itself is noise at this scale. "Changing only the random seed moves the score by 7.7 points, more than every data and recipe effect we measured," Ayoub Kirouane and Christos Petrocheilos write in a preprint posted to arXiv. "That null is our first result."

The study tested three MoE models from Alibaba, OpenAI and NVIDIA, each with 3.6 to 4.0 billion active parameters. Base models "never think in Greek: 0 of 1,000 reasoning traces, even when the question is Greek," the abstract reports, meaning the model answered correctly while reasoning in a form the user "cannot read, audit, or correct." After supervised fine-tuning, every released checkpoint reasoned in the language of the question on roughly 98% of items, with one family reaching that at 3x fewer tokens.

SFT alone did not clean up its own residues. About a quarter of answers skipped the requested format, answers leaked into the reasoning channel, and an explicit "think in English" instruction was obeyed under half the time. Reinforcement learning with verifiable rewards, pre-registered before training against a flat random-reward control, drove the format fallback from 24% to 2.5% and the leak rate from 3.5% to 0.0%, and moved English-obedience by 9.1 percentage points.

The authors propose six behavioural dimensions, each gated to reject any metric that correlates with output length, and disclose six failures their own instruments produced that only the controls caught. They release five checkpoints and say "the instruments, the controls and the pre-registration travel to any low-resource language; Greek is the case that let us measure them."