Paper turns NHTSA crash records into AV simulator tests
TL;DR
- Parashar and Fan describe an LLM pipeline that turns NHTSA accident records into runnable test scenarios for the MetaDrive driving simulator.
- A budget of just 20 generated scenarios was enough to expose interesting system failures in their experiments.
- Generated scenarios covered 4 road types, 3 vehicle movement patterns, and working-zone anomalies while staying within specified testing constraints.
A short paper posted to arXiv this week takes a different tack on one of the more expensive problems in autonomous driving: coming up with test scenarios that actually catch real failures instead of the ones engineers already know to look for.
The authors, Anjali Parashar and Chuchu Fan, describe a modular LLM-based pipeline that pulls categorical and contextual information out of NHTSA accident records written in natural language and then synthesizes matching scenarios for the MetaDrive simulator. In their experiments, the generated scenarios covered 4 road types, 3 vehicle movement patterns, and working-zone anomalies, and a budget of just 20 test scenarios was enough to surface interesting system failures.
Why this matters if you are not building an AV yourself: safety testing for self-driving systems is one of the places where the industry has long said it needs more scenarios, but hand-writing them is slow and mathematical sampling of a driving simulator tends to hit the same near-misses over and over. Real crash records are the closest thing to a distribution of what has actually gone wrong on public roads. Turning them into runnable simulator scenarios, cheaply and at volume, closes a loop that has stayed open for a while.
The honest caveat is the scale. A twenty-scenario demonstration on one simulator, from a two-author preprint, is a proof of concept, not evidence that this replaces the mixed regime of real-world miles, staged tests, and adversarial fuzzing that AV teams already run. The reporting doesn't give you a head-to-head against hand-authored scenarios on the same system, or an indication of how well the technique transfers to a different simulator or a production stack.
The direction is the part worth watching. If a language model can read a corpus of real failures and turn them into executable tests, the same idea extends beyond driving to robotics, industrial control, and any safety-critical system where incident reports pile up faster than the test suite grows.
Shared on Bluesky by 2 AI experts
Originally reported by arxiv.org
Read the original article →Original headline: Scenario Generation for Testing of Autonomous Driving Systems Using Real-World Failure Records