NCP-Bench: GPT-5.2 keeps 42% of stories intact at 20 turns

Interactive-storytelling research usually chases fluency; a new arXiv paper asks whether the story itself survives contact with the player. The team introduces NCP-Bench, short for Narrative Commitment Preservation, a set of 100 narrative environments derived from movie synopses. Each environment ships with a structured spec of trajectory, commitments, and initial facts, which lets an automated checker score whether a player agent and a narrator agent stay logically consistent turn after turn.

The headline number is that even the strongest model tested, GPT-5.2, maintains only a 42% survival rate after 20 turns. Across the six state-of-the-art systems the paper runs through the benchmark (GPT-5.2, GPT-4o-mini, DeepSeek-V3.2, Qwen3-235B-A22B, Kimi-K2.5, and Grok-4.1-Fast), fact conflicts dominate failures at 40% to 68%. The authors frame it as a gap between fluent prose and preserved state: models write cleanly and still contradict a fact they themselves established a few turns earlier once a user pushes back.

For anyone shipping an LLM-driven game, tutor, or long-lived companion, fluency benchmarks flatter these systems in exactly the wrong place. A model that scores well on turn-level coherence can still forget who is holding the knife by turn 20, and no amount of prompt scaffolding papers over fact-conflict rates that start north of 40%. Two experts in our Who's Who directory circulated the paper on the day it appeared, which tracks: this is the failure mode narrative-AI teams have been fighting privately for a year.

The scope is narrow by design. NCP-Bench runs on movie-synopsis worlds rather than full game engines, and the paper as summarised does not report how much of the gap closes with retrieval, structured memory, or tool-use scaffolds bolted on outside the base model. The six-model sample is a snapshot rather than a leaderboard, and the checker's own error bars around those survival and conflict numbers are not something to lean on hard.

Still, for product teams the near-term win is probably not a bigger frontier model, it is the plumbing around one: canonical fact stores, per-turn consistency checks, explicit commitment tracking. NCP-Bench at least gives that plumbing something concrete to optimize against.