STEB benchmark unifies style-embedding tests on 96 datasets
TL;DR
- STEB bundles 96 datasets across 7 languages to standardize how style text embeddings are evaluated.
- The paper finds semantic embeddings 'consistently fail' at stylistic tasks like authorship verification and AI-text detection.
- No single style embedding was universally best across STEB's tasks, which include authorship retrieval and linguistic-feature probing.
Semantic embeddings 'consistently fail in stylistic tasks,' according to a new arXiv paper posted June 30, 2026 by Rafael Rivera Soto, Anna Wegmann and Cristina Aggazzotti.
The paper introduces STEB, the Style Text Embedding Benchmark, which the authors frame as a response to a methodological gap: 'while semantic embeddings are rigorously evaluated on the Massive Text Embedding Benchmark, the evaluation of style embeddings remains fragmented, with each work relying on their own set of tasks and datasets.' STEB bundles 96 datasets across 7 languages, covering authorship verification, authorship retrieval, AI-text detection, and probing of linguistic features.
Their second headline finding is equally flat: 'there is no style embedding that is universally superior across all tasks evaluated.' The code base is open-sourced on GitHub.
No specific models or per-task numbers appear in the abstract. Two researchers we follow had already circulated the paper on the day it dropped.
Shared on Bluesky by 2 AI experts
Originally reported by arxiv.org
Read the original article →Original headline: STEB: Style Text Embedding Benchmark