NOLLI benchmark pins Korean LLM gap to writing-system tasks

A new arXiv paper from a group of Korean researchers takes a scalpel to the assumption that large language models simply have a broad Korean problem. The team behind NOLLI, a procedurally generated puzzle benchmark, argues the gap is much narrower than the usual leaderboard framing suggests, and that most of the trouble sits in a specific place: writing-system-intensive tasks over Hangul jamo, the sub-syllabic letters that compose Korean characters.

The setup is 15 puzzle types across 25 tasks and 7,500 items, with every instance seed-regenerable, verified to have a unique solution, and scored deterministically. Difficulty is calibrated behaviorally, meaning each generator is tuned until a fixed reference model lands in a target accuracy band, rather than assumed to track input size. Across 15 frontier, open-weight and Korean-developed models, 12 cleared a 3% overall-accuracy floor. For those 12, matched English and Korean accuracy came out statistically equivalent within a ±10 pp margin under a TOST equivalence test. That is the headline finding, and it cuts against the common story that Korean performance is broadly degraded relative to English.

Where the models do fall over is narrower. On a task the authors call Korean Cipher, English outperformed Korean by up to 68.7 percentage points. On Cryptarithmetic run over the same jamo characters, there was no systematic penalty. Jamo Composition accuracy predicted Korean Cipher accuracy, which the paper reads as consistent with difficulty in multi-step sub-syllabic execution rather than with the language itself. A separate Kinship deficit showed up positive in all 12 models. And a structural size measure failed to grow from Easy to Hard in 7 of 15 types, undercutting the habit of proxying difficulty by input length.

The honest caveat is one the authors flag themselves: these contrasts are diagnostic rather than causal, and NOLLI is puzzle-shaped, not a stand-in for real user tasks in Korean. What the paper does not settle is which of the tested models scored where, how the picture looks below the 3% floor, or whether the same pattern would hold on longer-form or agentic Korean workloads.

If most of the English-Korean gap really is concentrated in a small family of character-manipulation skills, the intuition that Korean models need broad, language-wide remediation may need to give way to something more surgical, aimed at jamo-level reasoning. That would be cheaper, more portable, and easier to verify with a seed-regenerable benchmark than another round of full-corpus retraining.