FLAWS benchmark shows top LLMs miss most planted paper errors

A new arXiv preprint puts a hard number against the drift toward using LLMs as peer reviewers, and it is a lot lower than the pitch decks have been implying. The benchmark, called FLAWS, is 713 paper-error pairs. The authors, Sarina Xi, Vishisht Rao, Justin Payan and Nihar B. Shah, built it by taking peer-reviewed papers and using LLMs to systematically insert claim-invalidating errors, then designed an automated metric to check whether frontier models could pinpoint them.

The scoreboard is sobering. Five frontier systems were evaluated: Claude Sonnet 4.5, DeepSeek Reasoner v3.1, Gemini 2.5 Pro, GPT-5, and Grok 4. The best of them, GPT-5, hit 39.1% identification accuracy at k=10, meaning even when the model was allowed to nominate its ten most likely error candidates for each paper, it caught the planted flaw in fewer than four out of ten cases. The other four models scored below that.

Why this matters if you are not building review tooling: journals, conferences, and grant programmes are all quietly piloting LLM-assisted review, and the implicit sales pitch is that a good model can flag the kinds of methodological slips a tired human reviewer might miss. A 39.1% top-ten hit rate on planted errors is a very different story from that pitch. It suggests the useful deployment mode is a candidate-suggester feeding a human, not an autonomous checker whose green light means anything.

The honest caveats are worth sitting with. These are synthetic errors, inserted by an LLM, not the messy human mistakes real manuscripts contain, so absolute numbers may over or understate what a deployed reviewer would catch on wild data. What the abstract does not give you is how performance breaks down by field, or whether GPT-5's edge comes from stronger reasoning or just from k=10 letting it fire more shots. Those are the questions to press on before anyone claims a milestone. What FLAWS does give the field is a shared, quantitative ceiling on error localisation, and it is markedly lower than the peer-review-by-LLM narrative has been suggesting.