'Code to Control' writes Python RL policies, skips LLM at runtime

The paper's offer is odd: use a language model to write a Python controller once, then throw the language model away at runtime. "Once learned, the resulting controllers require neither LLM inference nor planning at decision time, enabling real-time gameplay and faster action selection than a PPO policy," authors Zergham Ahmed, Joshua B. Tenenbaum, Chris Bates, and Samuel J. Gershman write in the arxiv preprint.

The approach splits the work. The LLM proposes the program structure. A derivative-free search then fits its parameters by interacting with the environment. "Code to Control learns from interaction and reward rather than relying on a task-specific natural-language command and predefined control primitives," the paper states, a jab at a strand of prior work where the LLM is handed a task description and a library of pre-built skills.

Latency is the stated motivation. "Recent LLM-based approaches to control either invoke a language model to select actions or synthesize world models that require planning at every decision, introducing latency that can limit real-time use," the authors write. Push the LLM's work into code generation and every runtime decision becomes a plain Python function call.

Benchmarks run across Atari, Flappy Bird, and MuJoCo locomotion. The paper claims the method "outperforms planning-based program synthesis methods, remains competitive with deep reinforcement learning while using fewer environment interactions, transfers across substantial changes in environment dynamics, and scales to complex locomotion tasks." The abstract publishes no per-game scores and no interaction counts; the comparisons sit as summary claims pending the full PDF. Two of the researchers we follow posted the paper link.