Modern LLMs are incredibly good compression algorithms, which can shed light on why autonomous data science agents don't overfit as much as you might think. arxiv.org/abs/2606.11045
- ML strategies that generalize well can be described in very few tokens, Bertran, Roth, and Wu argue.
- A reproducer agent given only a brief prompt successfully replicated high-performance models found by a full exploration agent.
- The framework was tested across 8 datasets covering tabular, vision, language, diffusion, and reward modeling tasks.