Python solver lifts Qwen2.5-32B to 90.53% on MedCalc-Bench
TL;DR
- On MedCalc-Bench Verified, Qwen2.5-32B-AWQ moved from 83.47% to 90.53% when arithmetic was handed off to a restricted Python executor.
- The same solver route at 7B gave 75.31% versus 72.02% direct, a paired +3.29 points with a confidence interval crossing zero.
- The authors audited the benchmark and flagged 16 of 55 calculators with version, use or coefficient concerns against current clinical guidelines.
"Large language models are unreliable at arithmetic, which is a problem for clinical calculators where a single numerical error changes the recommendation." That is the opening line of a paper on arXiv proposing what its authors call a Program-Solve interface: "the model does not calculate. Instead, it writes case-specific Python that a restricted local executor runs as a deterministic solver."
The team, led by Felipe Ocampo Osorio and including Leo Anthony Celi, tests it on MedCalc-Bench Verified, 1,100 cases across 55 calculators, using Qwen2.5-7B and Qwen2.5-32B-AWQ. Before running the models they audit the benchmark's formulas against current clinical guidelines and flag 16 of 55 "with version, use or coefficient concerns."
The results split by size. At 7B, handing math to the solver gives 75.31% against 72.02% for direct arithmetic, a paired +3.29 points with a 95% calculator-cluster interval of [-3.49, 10.38]. Not a reliable win. At 32B, the gap widens to 90.53% against 83.47%, +7.05 [0.47, 14.60], clear of zero.
A hand-written 22-calculator library the authors compare against is exact on its 440 supported cases but abstains elsewhere, covering 40.0% of the benchmark overall. Adding an executor, the authors conclude, "is not a substitute for verified formulas or reliable variable extraction either way."
Shared on Bluesky by 2 AI experts
Originally reported by arxiv.org
Read the original article →Original headline: Towards a Deterministic Math Solver for Clinical Language Models