PluraMath tests 27 LLMs on math in 18 low-resource languages

A new benchmark quietly re-opens a question the frontier-lab pitch has been treating as mostly settled. PluraMath, posted to arXiv this month, extends the earlier PolyMath math-reasoning dataset to cover 18 additional underrepresented languages spanning six language families, and it puts 27 reasoning LLMs through the same evaluation across four model scales, from small open models up to closed-source ensembles.

The finding the authors are willing to commit to is narrower than the scanner headlines might imply, but it is the interesting one. They report a persistent gap in mathematical reasoning performance between high-resource and underrepresented languages, and they specifically tie the stronger scores to better instruction-following ability rather than to raw parameter count. If your working assumption has been that the next flagship model simply eats this problem the way it ate English math benchmarks, this reads as counter-evidence in advance.

Method matters here because the market is full of translated benchmarks that are quietly broken by machine translation drift. The team went with a human-curated pipeline in which native speakers validated pre-computed translations, and open-sourced the dataset, the acquisition pipeline, and the evaluation framework under a CC BY-SA 4.0 license. That is the kind of release detail that decides whether a result gets reproduced by other groups or dies quietly on a poster.

The honest caveat is that the public arXiv summary does not itemize which 18 languages were added, which 27 models were tested, or the per-model score gaps, so any specific 'closed models still lose by X points on language Y' claim you may see downstream is not something the abstract itself supports. What you do get is a shape: the gap persists across scale, and instruction-tuning quality is what moves the needle. For teams building AI math tutors, translation tooling, or education products aimed at low-resource-language markets, that is a benchmark worth holding your vendors to before you ship, and worth watching as the authors and other groups extend the language list.