LLMs cover less of human writing than equal human samples do

When a large language model writes Harry Potter fanfiction, it tends to produce recognizable Hogwarts places and characters while missing the stylistically irregular content and relationship-diverse plotlines human fan writers bring. A new arxiv preprint from Zini Yang, Emily Wenger and Richard Jean So proposes two metrics for that gap: LLM Coverage (LLM-Cov) and In-Boundary Rate (IBR).

LLM-Cov measures how much of the human response distribution a model reaches on a given topic. IBR measures how often its samples fall inside the human-authored region. Across ideation and narrative tasks, the paper reports the tested LLMs had "lower mean coverage than equally sized held-out human samples," with model reach "disproportionately concentrated near the center of the human response space relative to this human benchmark."

The Harry Potter case gets a sharper finding. Covered and uncovered passages, the authors write, "show systematic differences in canon anchoring and narrative form." What the models land on versus what they skip, in other words, is not random; it tracks with how canonical or formally conventional a passage is.

The preprint's abstract lists no model names and no per-model coverage numbers. Two of the researchers on our radar had already circulated it by the time it hit the tracker.