SpyRL uses 'Who Is the Spy?' to reward open-ended LLM tasks

Reinforcement learning with verifiable rewards is a big part of why reasoning-focused LLMs have improved so quickly. It works on tasks where an answer is either right or wrong, math problems and code that has to pass tests, so a reward can be computed automatically at scale. It has not worked as well on open-ended writing, where the reward has to come from a human preference dataset, a reward model, or another LLM acting as judge. Each of those introduces bias, capability limits, and extra inference cost, in the framing of a new paper posted to arXiv.

The proposal is a training paradigm called RLSVR, for Reinforcement Learning with Self-Verifiable Rewards, and the specific method the authors build with it is SpyRL. The trick is to transform an open-ended task into what the authors describe as "verifiable proxy environments whose internal rules and interaction outcomes automatically generate reward signals."

The proxy environment is borrowed from the social deduction game Who Is the Spy?. Agents receive asymmetric information, complete the same target task, and vote to identify a designated spy. Because the spy identity is predetermined, whether the group votes correctly is a clean, checkable signal. The authors' claim is that this signal correlates with output quality, because doing the task well is what makes the spy easier for the others to catch.

The reported result across text summarization, creative writing, and mathematical reasoning is that SpyRL "outperforms existing self-improvement methods on non-verifiable tasks and yields consistent gains on verifiable reasoning tasks." If that generalizes, the same RL loop that pushed reasoning models forward has a route into open-ended domains that have been stuck on judges and preference models.

The honest caveats are worth being explicit about. The abstract does not name specific baselines, base models, or numeric deltas, so the size of the gain is not something a reader can size from the paper page alone. There is also a natural worry about self-play judging: a vote signal could end up rewarding outputs that are merely distinctive rather than better, and whether human raters actually agree with the spy signal is not addressed in the abstract.

If the method transfers as described, the more interesting practical implication is that smaller research groups get a candidate way to run RL on writing and summarization without needing a hand-labeled reward model. That is the direction worth watching over the next few months.