Pythia study: hidden states encode tasks before readout shows

At 0.4 to 1.1 percent of the way through Pythia's pretraining, the unembedding matrix reorganizes and WordNet word categories become decodable from its rows, according to a new preprint on arXiv. Before that narrow window, linear probes on the hidden states can already decode task information that the model's own outputs do not yet show.

The implication, as the authors put it, is that "timelines read from outputs conflate what the model encodes with what its readout expresses." A behavioral milestone, in other words, may be marking the moment the readout learned to speak the model's mind rather than the moment the model learned the task.

To track the readout directly, Matteo He, William F. Shen, Alex Iacob, Andrej Jovanovic, Xinchi Qiu and Nicholas D. Lane introduce what they call parameter-trajectory crosscoding, a method that fits "one sparse dictionary to every token's readout row at every checkpoint" and needs only checkpoint weights, not activations.

They then stress-test the gap by pretraining small models from scratch that differ only in the readout's learning rate. Raising it, the paper reports, "makes the readout develop earlier and the gap close sooner, with little effect on final loss." The abstract frames the finding squarely at the early-pretraining regime, "where many developmental transitions are dated," and does not extend the measurement to larger frontier systems.