UW-Madison's SAR Metric Tests How Compositional SAE Features Are

Learned sparse autoencoder coordinates behave more additively than random rotations of the same activation subspace, though pairwise interactions still carry predictive weight. In a steering benchmark on Llama-3.1 8B, modeling those pairs lifted Spearman rank fidelity from 0.5874 to 0.7790 and cut behavior-weighted ordering error from 0.1444 to 0.0606.

The result comes from a paper by Ziye Mei and Frederic Sala of the University of Wisconsin–Madison, which introduces a measure they call SAE Additivity Residuals, or SAR. The authors define it as "the fraction of behavioral variance unexplained by the best order r approximation". Across 18 model-and-behavior settings using Gemma-3 4B, Mistral-7B and Llama-3.1 8B on simple copying, subject verb agreement, gendered pronoun preference, CounterFact, IOI and greater-than comparison, the learned SAE basis showed "lower first order residuals for the learned coordinates across all 18 model and behavior settings" compared with sum-preserving random rotations of the same selected subspace.

The features aren't fully additive, though. "Pairwise models improve rank fidelity in all 18 model and behavior combinations, raising the average from 0.9782 to 0.9959," the authors report, with selection regret falling from 0.0272 to 0.0051. Pairwise models also beat parameter-matched controls that use randomly selected third-order terms.

The steering test applies the same framework to 16 published CorrSteer features. The authors sample 1,024 coalitions from the 2^16 possible combinations and score each on 1,400 MMLU-Pro questions drawn equally from the 14 official domains. Pairwise modeling "better predicts this aggregate steering response", with the held-out residual estimate dropping from 0.6424 to 0.3891. Code is released at the OpenSAE repository.