micha heilbron

Tracking entities is a key building block of understanding When measured as story comprehension rather than puzzle-solving, this ability emerges in models far smaller than previously thought -- and at scale, models far exceed human performance 9/9, fin arxiv.org/abs/2608.18083

Entity tracking emerges in sub-billion parameter language models and exceeds human performance in naturalistic narratives arxiv.org

Full, definitive paper here: direct.mit.edu/tacl/article...

direct.mit.edu

New preprint, w/ Karolina Drożdż Understanding language requires internally modelling what's happening where, even when unstated We built a new task to measure this ability in language models and people, and find it emerges at model scales far smaller than previously thought

Belated, but still happy to see our paper (with @drhanjones.bsky.social) on fleeting memory transformers is out in TACL! We find that giving language models human-like memory decay *improves* language learning, while, unexpectedly, impairing human reading time prediction Follow up results soon!