Berkenkamp team ties model-based RL to Lyapunov stability
TL;DR
- The paper extends Lyapunov stability verification and pairs it with statistical dynamics models to produce control policies with provable stability certificates.
- Under a Gaussian process prior, the authors prove that data collection itself can be done safely while expanding the certified safe region of the state space.
- In simulation, the method optimizes a neural network policy on an inverted pendulum without the pendulum ever falling down.
A 2017 NeurIPS paper by Felix Berkenkamp, Matteo Turchetta, Angela P. Schoellig and Andreas Krause proposes a reinforcement learning algorithm that will not knock over the pendulum it is learning to balance.
The authors start from a familiar complaint: "most reinforcement learning algorithms explore all possible actions, which may be harmful for real-world systems." Their fix is to import a tool from control theory. They "extend control-theoretic results on Lyapunov stability verification and show how to use statistical models of the dynamics to obtain high-performance control policies with provable stability certificates."
On top of that, with a Gaussian process prior on the dynamics, they claim to "prove that one can effectively and safely collect data in order to learn about the dynamics and thus both improve control performance and expand the safe region of the state space." The demo is narrow but literal: the method "can safely optimize a neural network policy on a simulated inverted pendulum, without the pendulum ever falling down."
The abstract reports no numbers on region size, sample complexity, or runtime, and the only experiment described is the simulated pendulum.
Shared on Bluesky by 1 AI expert
Originally reported by arxiv.org
Read the original article →Original headline: Safe Model-based Reinforcement Learning with Stability Guarantees