NYU's Wilson introduces 'epiplexity' to guide AI data selection

Natural language data carries measurably more structural information than images, according to a new framework introduced by NYU's Andrew Gordon Wilson and collaborators. In a talk posted on YouTube, Wilson argues that Shannon entropy and Kolmogorov complexity miss what modern deep learning actually extracts from data, and proposes "epiplexity" as a replacement measure aimed at computationally bounded observers.

Wilson's paper with Marc Finzi, Shikai Qiu, Yiding Jiang, Pavel Izmailov, and J. Zico Kolter frames the case against classical information theory as three paradoxes: that "deterministic transformations cannot increase information," that "information is order-independent," and that "likelihood modeling merely matches distributions." Modern practice contradicts all three.

The concrete comparison at the center of the claim: "natural language (like OpenWebText) has significantly higher epiplexity than image data (like CIFAR-5M) or chess data." The authors' interpretation is that "language data induces more complex, reusable structural information in models, potentially explaining why language pre-training leads to better general-purpose reasoning and out-of-distribution generalization."

A companion paper, "Epiplexity Guided Data Selection and Generation for Out-of-Distribution Generalization," puts numbers on the idea. On a 124M-parameter model, the group's EpiSelect procedure lifts average accuracy across ten LM Evaluation Harness tasks to 0.394 from a 0.379 baseline; on a 1.3B model, 0.431 versus 0.425. Epiplexity's correlation with out-of-distribution accuracy is reported at Pearson r=0.88. Two researchers we track shared the talk within our reading window.

One awkward finding sits inside the same paper: "simply training only on the PileCC domain leads to better performance than the state-of-the-art selection method." The authors read that as a limitation of The Pile as a benchmarking resource, not of their approach.