MLE-World 4B predicts ML experiment outcomes at 77.9% accuracy
TL;DR
- A 4B Qwen finetune called MLE-World predicts ML program outputs at 77.9% match on held-out code, versus 41.1% for larger prompted baselines.
- The predictor was trained on 497,056 agent-run programs across 17 MLE-bench Lite tasks, and runs 13–1,061× faster than prompted model baselines.
- Routed through a UCB search policy, it lifts an AIDE agent's mean percentile from 45.4 to 52.1 while cutting real executions 30–40%.
A four-billion-parameter finetune of Qwen3.5-4B called MLE-World can predict the terminal output of a machine-learning program at 77.9% match on held-out programs, versus 41.1% for much larger prompted models, according to a paper by Brian Ko, Han Wang, Hejia Geng, Li Erran Li and Sherry Yang. The predictor returns four fields in a single inference call: the exception class, the terminal output, the predicted validation metric, and a submission flag.
The training corpus was 497,056 program runs that AIDE-style research agents produced across 17 MLE-bench Lite tasks. "Its GPU inference time is 13–1,061× lower than that of the prompted models," the paper reports.
The authors wrap the predictor in five test-time-scaling policies and let an agent score several candidates by their predicted outputs before running anything. On the benchmark the UCB variant "reaches 52.1 mean percentile versus 45.4 for Base AIDE," and MLE-World-guided search achieves "up to 22.1% higher observed mean performance than the agent without a world model, with less real execution." Only candidates whose predicted outcome the model is unsure about get actually executed. "Gating saves 30–40% of real execution time while broadly maintaining performance," the paper notes.
The work lands amid a thick run of agents coverage; AI Weekly has logged 532 agent stories over the last 90 days.
Originally reported by huggingface.co
Read the original article →Original headline: MLE-World 4B Predicts ML Experiment Outcomes at 77.9% Accuracy, Cuts Agent Execution Time 30-40%