Bergemann, Koh, Morris recast AI alignment as mechanism design

Three economists have translated the AI alignment problem into the language of mechanism design, treating an AI agent as a party whose preferences and capabilities the principal cannot observe. In Mechanism Design for Alignment and Control, Dirk Bergemann, Andrew Koh and Stephen Morris develop a framework in which "mechanisms must incentivize both honesty and obedience," and derive a revelation principle for the setting.

The technical hinge is what the authors call a one-sided imitation structure: the assumption that "capabilities can be concealed but not counterfeited." That asymmetry, they argue, is what makes an alignment mechanism tractable, yielding "a characterization of implementable policies via nested cyclical monotonicity, and conditions under which eliciting higher-order beliefs can discipline multiple agents."

The paper walks through five stylized applications, including sandbagging (a more capable agent pretending to be less capable), peer scoring as a discipline device, and coupling rewards "to induce competition among multiple agents." The alignment and interpretability pair may be the most striking. The authors describe the two as "substitutes in the instrument but complements in value," a formal way of saying that a mechanism designer can trade one for the other while a user still wants both.

Three of the researchers we track on our Who's Who directory posted the preprint the day it appeared.