4B Qwen student edges GPT-5.6 on transit-kiosk benchmark

A 4-billion-parameter Qwen 3.5 student, fine-tuned with parameter-efficient methods and quantized to a 2.6 GB footprint, edged out both tiers of GPT-5.6 on a new transit-kiosk benchmark and matched GPT-5.4 at maximum reasoning effort. The arXiv paper introduces MetroLLM-Bench, a 955-case suite that treats a language model as the policy layer of a station kiosk, covering routing, fare calculation, disruptions, accessibility and adversarial input across six real metro systems ranging from 37 to 414 stations.

The headline result lives on Tier 1, the deterministic slice made of fourteen scoring components. "A 4B Qwen 3.5 student trained through parameter-efficient fine-tuning (PEFT) exceeds both GPT-5.6 tiers on Tier 1 (91.3 against 90.6 and 90.0) and matches GPT-5.4 full at maximum reasoning effort (91.4), with a 2.6 GB Q4_K_M footprint," the paper reports. A deterministic rule-based baseline still reaches 84.6 on Tier 1, with the remaining language-model advantage concentrated in "policy adaptation, compound scenarios, accessibility, and temporal reasoning."

The scaling story is the twist. "Larger 9B and 27B students provide no further Tier 1 improvement over the 4B student at this training scale," the paper states, and the PEFT gain over the corresponding base model flips direction as parameters grow: +7.03 points at 2B down to -0.91 at 27B, with every seed pointing the same way at every size.

On the composite ranking that folds in Tier 2's eight semantic-quality components, six of which are judged by another language model, Muse Glimmer 30B leads. Twenty-six models from six vendors were evaluated; twenty-three were ranked. Serving configuration alone moved a Qwen 3.5-to-3.8 comparison by 2.7 Tier 1 points, a reminder that inference plumbing is doing real work inside these numbers. The benchmark, harness and fine-tuned students are posted at github.com/continker/metrollm-bench.