doi.org web signal

Iceland Paper: Human Grammar Tests Misread LLMs, Authors Argue

By Published Updated

TL;DR

  • Elena Callegari and Annika Simonsen argue traditional linguistic tests probe human competence and should not be used that way on LLMs.
  • The paper leans on the analogy that birds and airplanes achieve flight 'through fundamentally different means' to argue similar outputs mask different mechanisms.
  • Performance-oriented tests can still measure LLM behavior meaningfully, the authors argue, but competence-probing tests do not transfer.

Traditional linguistic tests built to probe human competence should not be used to draw competence-level conclusions about large language models, two linguists at the University of Iceland argue in a new paper published October 2, 2026 in the Northern European Journal of Language Technology. Authors Elena Callegari and Annika Simonsen frame the case with the analogy that gives their work its title: birds and airplanes both fly, "both achieving flight through fundamentally different means," so "similar linguistic outputs do not imply shared underlying processes."

The argument turns on an older distinction between competence and performance. Tests that operationalize performance, the authors write, can still provide meaningful measures of LLM behavior; tests designed to probe human linguistic competence misread what they are measuring when applied to machines, which the paper says "do not acquire language through the same mechanisms and do not possess competence in the human sense."

The piece appeared in Volume 12, Number 1 of NEJLT. Two researchers we track posted the link within days of release.

Shared on Bluesky by 2 AI experts

Originally reported by doi.org

Read the original article →

Original headline: Rethinking Linguistic Tests for Large Language Models: Airplanes Aren't Birds | Northern European Journal of Language Technology