DrivingBench puts four frontier LLMs behind a real Corolla's wheel

Three Bay Area tech workers wired four frontier chatbots into the steering, throttle and brakes of a Toyota Corolla, drove it around a cone course in a parking lot, and are calling the result a benchmark. Only one model finished.

The team, Aditya Ramabadran, Tobias Gessler and Simon Mahns, all from an AI math startup called Axiom Math, built the rig using Comma, an open-source self-driving system, as 404 Media reports. Two cameras and telemetry fed a laptop, which prompted the LLM; the LLM's commands went back through Comma to the car. A human sat in the driver's seat the whole time. "And we have the foot over the brake, just in case," Mahns said.

The models refused to drive.

"We tried calling it a simulation, which worked like some percentage of the time, but then other times they would see the images and see, oh, 'I'm in a real parking lot, these people are just lying to me,'" Ramabadran told 404 Media. The workaround was to relabel the whole thing. "We ended up having to call everything a sandbox. And with that prompt, it's able to consistently drive the car and like never refuse to do that."

OpenAI's GPT-6 Astra finished the course on its second attempt, in about five minutes, at 0.94 mph. xAI's Grok 4.6, OpenAI's GPT-5.6 Sol and Anthropic's Claude Fable 5.1 each drove only a few meters. Mahns called it "an interesting demonstration of potential emergent capabilities," and conceded that when the team tried to get Astra to write the glue code, "it was pure slop."

Comma itself is under a National Highway Traffic Safety Administration investigation over two crashes that killed three people.