MLE-World 4B predicts ML experiment outcomes at 77.9% accuracy
TL;DR
- The 4B model hits 77.9% terminal output accuracy vs. 41.1% for prompted models 10x larger, plus 86.4% exception class accuracy and 78.1% validation metric accuracy.
- Five distinct test-time scaling policies (Sequential revision, Best-of-N, Pairwise judging, Fusion, UCB) all show gains, confirming the world model generalizes across search strategies.
- GPU inference runs at 2.7 GPU-minutes per call, a 13-1,061x cost advantage over prompted baselines that compounds at research-lab scale.
A four-billion-parameter finetune of Qwen3.5-4B called MLE-World can predict the terminal output of a machine-learning program at 77.9% match on held-out programs, versus 41.1% for much larger prompted models, according to a paper by Brian Ko, Han Wang, Hejia Geng, Li Erran Li and Sherry Yang. The predictor returns four fields in a single inference call: the exception class, the terminal output, the predicted validation metric, and a submission flag.
The training corpus was 497,056 program runs that AIDE-style research agents produced across 17 MLE-bench Lite tasks. "Its GPU inference time is 13–1,061× lower than that of the prompted models," the paper reports.
The authors wrap the predictor in five test-time-scaling policies and let an agent score several candidates by their predicted outputs before running anything. On the benchmark the UCB variant "reaches 52.1 mean percentile versus 45.4 for Base AIDE," and MLE-World-guided search achieves "up to 22.1% higher observed mean performance than the agent without a world model, with less real execution." Only candidates whose predicted outcome the model is unsure about get actually executed. "Gating saves 30–40% of real execution time while broadly maintaining performance," the paper notes.
The work lands amid a thick run of agents coverage; AI Weekly has logged 532 agent stories over the last 90 days.
What others are reporting
-
arXiv Read →
Canonical academic record; names the five-institution author team (Columbia, NYU, UIUC, Oxford, UW) and formalizes the CC BY 4.0 open release.
"MLE-World predicts a program's terminal output from its code and data description without executing it, together with a confidence score."
-
MLE-World Project Page Read →
First-party page details all five test-time scaling policies and adds separate accuracy figures for exception class (86.4%) and validation metric (78.1%) beyond the headline number.
"MLE-World predicts execution feedback directly from code and task context: terminal output, an exception class, and a validation metric."
-
Papers With Code Read →
Lists actionable release artifacts: model weights at ko120/MLE-World-Qwen3.5-4B on HuggingFace and code at ko120/mle_world on GitHub for immediate deployment.
Originally reported by huggingface.co
Read the original article →Original headline: MLE-World 4B Predicts ML Experiment Outcomes at 77.9% Accuracy, Cuts Agent Execution Time 30-40%