Language encoders track readability signals more clearly than XLM-R
TL;DR
- Wong and Tanner test whether transformers used for readability assessment internalize the same features as traditional classifiers across five languages.
- Language-specific encoders track traditional models more clearly than multilingual XLM-R, with alignment varying by model family, language, and layer.
- The paper warns that high linear separability does not always imply directional influence, limiting linear probing for count-based readability features.
Transformers used for automatic readability assessment recover the surface-length, syntactic, and lexical-diversity signals of traditional feature-based classifiers, but the alignment varies by model family, language, and layer, according to a paper on arxiv by Joshua Wong and Chris Tanner.
Testing across Arabic, English, French, Hindi, and Russian on the ReadMe++ dataset, the authors used SHAP to identify the features driving traditional classifiers, then applied those as TCAV concept sets to probe multilingual XLM-R and language-specific encoders. The paper reports "language-specific encoders tracking traditional models more clearly than XLM-R."
The authors flag an interpretability caveat: "High linear separability does not always imply directional influence, limiting linear probing for count-based readability features." They emphasize that readability labels are subjective and rater-dependent, so "high accuracy on noisy ground truth may reflect surface patterns rather than the linguistic structure that defines difficulty."
Shared on Bluesky by 2 AI experts
Originally reported by arxiv.org
Read the original article →Original headline: Analyzing Traditional and Neural Approaches to Multilingual Readability Assessment