Rajpal, Goodman tie compression bottleneck to RNN generalization

Something worth flagging on arxiv about what actually makes a neural network generalise versus memorise, with an unusual side check against mouse brains. In a new paper on arXiv, Hardik Rajpal and Dan Goodman argue that an explicit information bottleneck forcing a recurrent network to learn a low-dimensional representation is necessary for rotational and out-of-distribution generalisation on a time-series prediction task. Not helpful, necessary.

The unusual bit is what happens during training. Using information-theoretic measures of causal emergence, they characterise the representation across the memorisation-to-generalisation transition and find it does not grow smoothly. It follows a non-monotonic trajectory: an initial decrease, a minimum, and a subsequent rise to a maximum, even as prediction loss falls monotonically. The magnitude of that emergent structure, the paper claims, reliably predicts generalisation performance, and the trajectory scales with task complexity.

The more surprising move is checking the same signature in biology. In CA1 hippocampal activity in mice learning an alternating maze task, the authors report analogous non-monotonic emergence dynamics that track behavioural performance, with similar behaviour observed in medial PFC. Same shape in the model, same shape in the animal. Their read is that this supports a causal role for learned, compact, distributed representations in cognition, rather than the compressed structure being merely an artifact of dimensionality reduction.

Why this matters if you are not doing neuroscience: a lot of current framing treats scale plus data as the main driver of capability, with interpretability of internal representations as a nice-to-have. A finding that an explicit compression constraint is necessary for a specific kind of generalisation pushes back on that, and hands regularisation and evaluation teams a candidate metric to watch mid-training rather than only at the end.

The honest caveat is that the demonstration is on a recurrent network doing time-series prediction, not a transformer language model at frontier scale, and causal emergence measures are still contested in the information-theory literature. What the paper does not tell you is whether the same signature survives in larger, differently structured models, or how sensitive the bottleneck design is across tasks. Even so, if the shape of learning carries an observable signal you can measure while training is still running, that is the sort of thing interpretability and evaluation groups will want to test.