MMDM Benchmark: 28 Decision Models, Multi-Select Caps at 37%

WeMM-JEV 27B tops MMDM's hard single-select track at 76.00%, but no model in the 28-system field clears 36.81% on hard multi-select — and every one of them scores below 0.20 on the benchmark's Separation-Balance F metric for recovering a full correct set. The authors introduce MMDM in a Hugging Face paper listing as 5,448 questions across 16 task categories, with option spaces from 2 up to 1,024 and inputs that stretch to 64,817 tokens.

The design targets what prior benchmarks miss. The abstract frames it as capturing "long contexts, large option spaces, and complex logical dependencies, challenges that existing benchmarks do not adequately capture together." System-1 decision models, in the authors' definition, "directly make structured probabilistic decisions over query-specified options given a state in a single forward pass, without autoregressively generating reasoning trajectories or textual responses."

What jumps out of the leaderboard is how little the aggregate ranking survives slicing. OpenJev beats Intern-Decision on every option-count bin until inputs pass 8K tokens, where the order flips to 15.5% vs 34.5%. On option sets of 129 to 1,024, OpenJev reverses it again, 26.0% to 8.3%. Jev and Microsoft Decision-1 sit within about 7 points of each other on 2-to-4-option questions and diverge by more than 37 points on the largest option bin. The highest multi-select SB-F anyone posts is 0.1905, from WeMM-JEV 27B.

MMDM lands into a crowded week for agent-evaluation work that our tracker has been logging — 569 agent stories in the last 90 days, including yesterday's WikiTrace paper on memory-induced regression. The paper publishes no per-model multi-select breakdown for every system in the 28-model field, and the SB-F ceiling is reported in aggregate rather than tied to a specific vendor's architecture choices.