Mechanistic Topic Models trade word lists for SAE features

Topic models usually describe a theme by listing its most frequent words. A new paper in the Transactions of the Association for Computational Linguistics proposes defining topics as directions in a sparse autoencoder's feature space instead.

The paper, "Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders," is by Carolina Zheng, Nicolas Beltran-Velez, Sweta Karlekar, Claudia Shi, Achille Nazaret, Asif Mallik, Amir Feder and David M. Blei, and runs pages 2187–2212 of TACL volume 14. The authors call the method Mechanistic Topic Models, or MTMs. "By defining topics over this semantically rich space, MTMs can reveal deeper conceptual themes with expressive feature descriptions," they write.

The second claim is operational: "uniquely among topic models, MTMs enable controllable text generation using topic steering vectors." In other words the same object that labels a corpus can also push an LLM's output toward a given theme.

To compare MTM topics against classical word-list topics the authors introduce "topic judge, an LLM-based pairwise comparison evaluation framework," and run it on eight datasets. The reported result: MTMs "match or exceed traditional and neural baselines on coherence metrics, are consistently preferred by topic judge, and enable effective LLM steering."

The abstract does not name the base language model, the specific SAE used, the eight datasets, or per-benchmark numbers.