StoryScope detects AI fiction by narrative shape, not style

A new paper posted to arXiv approaches AI-written fiction detection by looking past writing style at the shape of the story itself. Instead of surface tells that models can paper over with each release, the authors focus on what they call discourse-level narrative choices: character agency, chronological structure, that kind of thing.

The system, StoryScope, was built and tested on 61,608 stories of roughly 5,000 words each, drawn from 10,272 writing prompts where each prompt was written by one human and five different LLMs. Using narrative features alone, the classifier hit 93.2% macro-F1 for human-versus-AI detection. On the harder six-way task of naming which model wrote a story, it hit 68.4%. Both scores kept over 97% of the performance of a version that also uses stylistic cues, which is the punchline: the narrative fingerprint is doing almost all the work.

The specifics of the fingerprint are the interesting part. AI stories, the authors report, 'over-explain themes and favor tidy, single-track plots', while human stories frame protagonist choices as more morally ambiguous and show more temporal complexity. Individual models have their own tells: Claude shows flat event escalation, GPT over-indexes on dream sequences, Gemini defaults to external character description. AI-generated stories cluster together in the feature space; human ones are much more diverse.

Take the specifics as reported, not settled. This is one paper on writing-prompt fiction of a bounded length, and the moment a detector's tells are public they become a training target: a model told to stop leaning on dream sequences can probably learn to stop. What the reporting doesn't give you is whether these narrative patterns hold up in longer forms like novels, or in genres where tidy plotting is a human convention rather than a giveaway.

Still, the direction is the part worth watching. For publishers, universities, and writing platforms trying to keep the slush pile honest, a detector that reads plot rather than prose is harder to defeat with a rewrite pass. And for the model builders, the paper doubles as a shopping list of what to fix if 'writes like a human' is actually the goal.