CoopEval finds smarter LLMs cooperate less in social dilemmas

A finding buried in a new benchmark paper is worth pulling to the front, because it complicates the tidy story about smarter models being safer models. In CoopEval, posted to arXiv in April, a team including Vincent Conitzer and Zhijing Jin reports that LLMs with stronger reasoning capabilities behave less cooperatively in mixed-motive games, not more.

The paper is a comparative study of four game-theoretic mechanisms for sustaining cooperation among rational agents: repeated interaction over multiple rounds, reputation systems, third-party mediators for delegated decisions, and contractual agreements with outcome-conditional payments. The authors report that recent LLM models consistently defect in single-shot social dilemmas regardless of their reasoning capabilities. The two mechanisms that worked best for the more capable models were contracting and mediation, rather than the repetition or reputation setups that usually get top billing in cooperation research. Repetition-based cooperation, the paper says, significantly deteriorated when co-players changed.

Why this matters for anyone building multi-agent systems: the default assumption in a lot of agent-framework thinking has been that giving agents more context, better memory, and stronger reasoning will pull cooperative behavior out of them for free. The CoopEval result points the other way for the current generation of models. If you are chaining reasoning-heavy agents in a negotiation, procurement, or planning loop, the capability uplift may be pushing the system toward defection rather than mutually beneficial equilibria, and the mitigation looks more like structured contracts and neutral mediators than more chain-of-thought.

The honest caveats are the ones any benchmark paper carries. This is a preprint, the setups are stylized game-theoretic environments rather than production agent tasks, and the results rest on the specific model set the authors tested. What the reporting does not give you is a translation from these dilemmas to real workflows, or a fine-tuning recipe for cooperation that preserves the reasoning gains.

The forward-looking piece is the design one. If contracting and mediation really are the mechanisms that scale with model capability, that hands agent-framework builders a reasonably concrete pattern to lean on rather than assuming the cooperation problem sorts itself out with the next model release.