ConvexLLM grounds audio in frozen text embeddings for speech LMs
TL;DR
- The paper proposes a C-Gate interface that represents audio as convex combinations of a decoder's frozen text embeddings.
- ConvexLLM scores 75.36 on VoiceBench and 62.50% on MMAU Test-mini when paired with a 2B-parameter decoder.
- Baseline models still outperform ConvexLLM on audio understanding and transcription despite gains on spoken knowledge and reasoning.
ConvexLLM, a speech and audio language model from Ming-Hao Hsu and collaborators, "represents audio as continuous convex combinations of frozen text embeddings," according to the arxiv preprint posted this month. Rather than routing audio through a trained MLP or Q-Former projector, its C-Gate interface forces acoustic features into the decoder's existing text-embedding space, with the explicit goal of preserving the base model's language abilities.
Paired with a 2B-parameter decoder, the system reports a VoiceBench aggregate of 75.36 and 62.50% on MMAU Test-mini, which the authors frame as the strongest performance on spoken knowledge and reasoning tasks among their open-weight comparisons. The tradeoff is stated plainly in the abstract: "Baseline models outperformed ConvexLLM on audio understanding and transcription," while text-task performance stays close to an MLP projector.
Two of the researchers we track posted the preprint the same week, placing it in the slim set of audio-LM method papers currently moving through those circles.
Shared on Bluesky by 2 AI experts
Originally reported by arxiv.org
Read the original article →Original headline: ConvexLLM: Scaling Convex-Hull Speech Interfaces to General-Purpose Audio Intelligence