Amato publishes primer on cooperative multi-agent RL methods

Christopher Amato has posted a tutorial paper on cooperative multi-agent reinforcement learning, organizing the field around three settings: centralized training and execution (CTE), centralized training for decentralized execution (CTDE), and decentralized training and execution (DTE).

The text walks through the main methods in each camp, covering independent Q-learning and policy gradient extensions, value factorization approaches including VDN, QMIX and QPLEX, and centralized critic methods including MADDPG, COMA and MAPPO. Amato writes that MARL "has exploded in popularity in recent years" and that CTDE methods "are the most common, as they leverage centralized information during training while enabling decentralized execution."

The scope is admittedly partial. "It does not cover all work in cooperative MARL as the area is quite extensive," Amato writes, adding that the paper also discusses "common misconceptions, the relationship between different approaches, and some open questions." Two of the researchers on our who's-who tracker circulated the arxiv link.