Galbot: Public Scores Miss How Fast VLMs Self-Improve on Robots

The averaged public-benchmark score does not predict how fast a vision-language model self-improves on a robot. "The five-axis metrics also distinguish systems with close public benchmark averages and show that high scores need not come with early or efficient gains," reads the abstract of a paper posted October 19 on Hugging Face by Zekai Li, Zekun Qi and nine co-authors from the Galbot Team.

Their framework, PhysRSI-eval, scores agents along five axes (First-Attempt Performance, Efficiency, Success, Budget and Improvement from failure) across 18 manipulation tasks under a fixed interaction budget. The project page lists GPT-6 Astra, GPT-6.1 Sol, GPT-6 Sol, GPT-5.6 Sol, Claude Opus 5.5, Claude Sonnet 5.5, and a separate run with Gemini 4.

Out of those measurements comes the Physical Barrel Hypothesis: "different capabilities may constrain different parts of improvement." The weakest axis caps the ceiling.

Workspace interventions produced a parallel result. The abstract reports that "frozen workspaces remain useful on unseen layouts, and three cross-task cases achieve success earlier," which the authors read as evidence that "scene-specific adaptation and reusable experience can coexist."

The paper lands in a busy week for robotics evaluation. Our tracker is up to 167 robotics stories in the last 90 days, with RoboValue and HandsOn from Tsinghua both landing yesterday.