MIT and NYU Build FLEX: Robots Read Intent Behind Commands

A robot is carrying a cup of coffee across a cluttered table when the person types "Don't spill on the laptop!" The robot, instead of just avoiding that one laptop, reasons about why it should avoid the laptop at all.

That is the setup for FLEX, a system from researchers at MIT and NYU described in a post on MIT News. The pipeline runs in three steps, Add, Generalize, and Mask, and uses GPT-4o to convert a natural-language correction into a broader feature the robot's motion planner can reuse. The whole cycle takes about five seconds.

The problem the authors want to solve is that home environments are not labeled. "Everyone's homes are different, and everyone cares about different things, so no robot comes out of the box knowing what matters to you," says Andreea Bobu, assistant professor of aeronautics and astronautics at MIT, who leads the Collaborative Learning and Autonomy Research (CLEAR) Lab. Bobu also notes that robots use math to do their trajectory planning, and the team cannot expect a consumer to learn that math in order to tell a robot what to do.

FLEX's bet is that the common sense baked into a big LLM can fill that gap. "LLMs have seen so many texts from the internet. People might have written about, 'Oh, my phone dropped in the water and now it doesn't work,' or 'I spilled coffee on my computer and now it doesn't work,'" says Nathan Dennler, a postdoctoral researcher at MIT CSAIL. "Because they've trained so much on those things, some of that common sense gets absorbed into the LLM." The Generalize step is where GPT-4o scans those patterns and promotes a one-off correction into a rule about features like liquid-near-device.

The team tested FLEX in simulation on three tasks: serving coffee, moving a candle, and carrying a bowling ball. The MIT write-up says the robot "followed all three commands literally and also reasoned at a higher level," but publishes no head-to-head reward numbers or baseline comparisons in the piece itself. The paper, "Flexible Language-Guided Representation Editing for Adaptive Robot Behavior," is slated for the Conference on Robot Learning this November; it lands in a steady stream of lab work our feed has logged across 194 robotics stories in the past 90 days.

"What excites me about FLEX is that it can pick that up from a sentence or two instead of asking you to spell out every rule," Bobu says. The work was funded by the MIT Generative AI Impact Consortium and MIT Schwarzman College of Computing's Social and Ethical Responsibilities program.