arXiv paper finds no universally superior discovery harness

There is a running assumption in AI research that once you have a good autonomous discovery system, you plug it in and let it work. A new paper on arXiv argues that is a mistake. Systems like OpenEvolve and TTT-Discover are described there not as single tools but as composites of decisions about archives, parent selection, exploration, and budget allocation, and the central finding is that no fixed combination wins across the tasks tested.

The authors, Akshat Gupta, Jermaine Lei, Alexander Lu, Gopala Anumanchipalli and Leshem Choshen, ran 30 budget-matched harnesses across 12 model-problem pairs, burning through more than 3.1 million LLM rollouts to see whether a universal recipe would emerge. It did not. Their headline line is that no fixed harness is reliably superior across the evaluated model-problem pairs, and variants of OpenEvolve in particular often underperformed simpler alternatives.

The more interesting turn is what worked instead. An adaptive-allocation experiment that starts several harnesses at once, prunes the weaker ones, and rolls the leftover budget into the stronger candidates beat both committing to a single harness and running a non-adaptive ensemble. That reframes discovery infrastructure as something you tune per problem rather than pick once.

The honest caveat is that this is one paper testing 12 model-problem pairs, and the summary does not break down which harnesses win where, or what the adaptive orchestration itself costs relative to the compute it saves. Its practical contribution beyond the finding is a released run pool and baseline distributions, which give other researchers reusable statistical scaffolding to check their own harnesses against. For teams currently standardizing on one discovery system, the immediate move is to treat that choice as a hyperparameter, not a default.