Encoding Probe reconstructs transformer states from features

Flipping the direction of a standard interpretability probe shows that speaker-related effects in transformers 'vary strongly across different training objectives and datasets,' while syntactic and lexical features 'contribute independently to reconstruction.'

The result comes from an EMNLP 2026 paper by Gaofei Shen, Martijn Bentum, Tomas Ostar Lentz, Afra Alishahi, and Grzegorz Chrupała. The paper introduces what the authors call an Encoding Probe. Where the common decoding probe asks what can be read out of a model's internal state, the Encoding Probe reconstructs that internal state from a chosen set of interpretable features and measures how much each contributes.

The authors argue the reversal fixes two problems in standard probing: 'contributions of different features to model representations cannot be directly compared, and feature correlations can affect probing results.' They evaluate the method on text and speech transformer models, with feature sets 'spanning acoustics, phonetics, syntax, lexicon, and speaker identity.'

Two of the researchers we follow in our Who's Who directory shared the paper this week. The authors frame the Encoding Probe as 'a complementary perspective on interpreting model representations beyond decodability.' The abstract names no specific text or speech model, and publishes no per-feature reconstruction scores.