huggingface.co web signal

InclusionMed ranks 22 medical AIs; Gemini 4 Argon tops at 49.1

TL;DR

  • Of 388 candidate medical AI benchmarks, InclusionMed kept four public plus one private, covering just three of nine role-capability cells.
  • Across 22 frontier models the Index spanned 17.5 to 49.1 of 100, with Gemini 4 Argon leading and Opus 5.5 and Fable 5.1 tied at 48.9.
  • Kimi K3 and GPT-5.6 Luna scored 37.9 and 36.9 but cost $1.51 vs $0.064 per task run, a roughly 24-fold price gap for near-equal performance.

Of 388 candidate medical AI benchmarks pulled from the literature, public repositories, and community contributions, InclusionMed kept four. The rest washed out through a six-criterion audit and independent reproduction; three medical-knowledge benchmarks dropped out for score saturation, and the broader HealthBench was retained only as its Professional subset after limited discrimination among models on the full set.

The four that survived were HealthBench Professional, PhysicianBench, the Health and Medicine subset of ALE-CLI v1.0, and CHI-Bench, joined by InclusionMed-DxFiles, a private diagnostic-reasoning benchmark the authors built. Together the five populate three of the nine cells in an Atlas that pairs three healthcare roles (patients and the public, clinicians, administrators) with three AI capabilities (knowledge, reasoning, agency).

Twenty-two frontier models were run against the portfolio. "Overall InclusionMed Index scores range from 17.5 to 49.1 on the 0–100 scale." Gemini 4 Argon led at 49.1, followed by Claude Opus 5.5 and Claude Fable 5.1 (both 48.9), with GPT-6 Astra (45.1) and GPT-6.1 Sol (44.5) rounding out the top five.

Rank depended heavily on which slice you asked for. "Claude Opus 5.5 leads for healthcare professionals (55.6), GPT-6 Astra leads in reasoning (58.7), and Claude Fable 5.1 leads in agency (46.1)." GLM-5.3 ranked 14th for clinicians but 4th for administrators; GPT-6 Luna slid from 8th in reasoning to 20th in agency.

On HealthBench Professional, every one of the top ten models scored higher on patient communication (median 69.1) than on clinical decision-making. Medical calculation (47.0) and safety and ethics (48.8) were the lowest expertise medians; treatment planning and implementation had the lowest workflow median at 53.0; psychiatry had the lowest specialty median at 49.4, with pediatrics (49.9) close behind.

Price did not track performance. Claude Fable 5.1 and Claude Opus 5.5 both rounded to 48.9, yet "Fable's estimated cost per task run was approximately 2.5 times that of Opus ($4.12 versus $1.68)." The sharper comparison sat lower in the table: Kimi K3 at 37.9 and GPT-5.6 Luna at 36.9, costing $1.51 and $0.064 per task run. "At these rates, a fixed inference budget could support approximately 24 times as many task runs with GPT-5.6 Luna as with Kimi K3 for the same benchmark mix."

The authors also flag what their Atlas does not yet cover. No current benchmark targets patients and the public, and of 1,034 tagged tasks only 338 had a determinable geographic setting, with 318 of those tied to the United States.