SO-Arena scores 21 System One models on 2,919 tasks

The best-performing System One model on NUS's new SO-Arena benchmark ranks only seventh when the same models are scored for decision reliability. Capability and reliability come apart.

The paper, posted on Hugging Face on 13 November 2026, scores 21 models on 2,919 tasks drawn from 11 domains, grouped into five capability families: semantic understanding, knowledge and reasoning, safety and policy compliance, evidence assessment, and agent task completion. Among the evaluated systems are 16 open-weight System One models, one auto-switching model, two closed APIs, and two System Two references (GPT-6 Sol and GPT-6 Luna).

At the top of the capability table, Fastino's GLiDE (27B) and PelaAI's KnowLine-31B tie at 70.2. GPT-6 Sol posts 82.3 and GPT-6 Luna 79.7 as the reference System Two numbers. SimpleJev's JevAny-27B is called out as the most reliable System One model despite ranking eighth in capability, while TypeSafe's Jev API 1.14 is ranked sixth in capability but named among the most reliable.

"The best-performing System One model ranks only seventh in reliability," the authors write, arguing that "strong overall performance does not necessarily imply reliable decision-making."

Reliability itself is split in two. Option Order Stability measures how often a model's answer survives 12 reorderings of the choices. Null Discrimination tests whether the model notices when the correct option is absent. Fourteen of the 18 System One models score above 0.8 on order stability, but on null discrimination the paper reports a range from near chance to roughly 0.8, with the authors noting that "a performance score therefore says much about a model's order stability but little about whether it notices a missing answer."

The paper separates domains into those where System One matches System Two (knowledge, sentiment, retrieval), those where it lands within a few points (intent routing, safety, tool use, web interaction), and those where both still trail (math, scientific reasoning, code). On math, the auto-switching StartLux Auto rises from 50.8 to 92.5 once thinking is enabled.

On speed, the authors claim that "every open-weight System One model answers within 0.3 seconds per task on one H100 GPU, tens of times faster than GPT-6 Sol and Luna." This lands in a busy week for the category; it is one of several open-weight releases we tracked on 13 November alongside work from IBM and Zyphra.