AVE-Compass benchmark exposes gaps in AI audio-video edits

Video editing evaluation has been sitting on a quiet assumption for a while, that you can grade a visual model on silent clips, grade an audio model on isolated waveforms, and treat the two as separate problems. A new arXiv paper introducing AVE-Compass pushes back on that, arguing most real edits are cross-modal and asking how current models cope when instructed to change both channels at once.

The benchmark's shape is worth stating plainly. It bundles 145 curated source videos with 196 audio-visually coupled editing instructions, expanded into 2,688 fine-grained checklist items, scored through a mix of multimodal large language model judging and automated metrics. Four axes get graded: Instruction Following, Fidelity Preserving, Realism, and Editing Intent. That last one matters, because a lot of editing failures are technically correct changes that miss what the user actually wanted.

The headline finding, in the authors' own words, is that state-of-the-art models still struggle to execute cross-modal instructions while preserving non-target content. Change the sound and other things drift; change the visual and the audio stops tracking. The paper also proposes AVE-Agent, a modular agent framework using task decomposition and self-reflection, and reports it lifted performance across multiple dimensions relative to the models it wraps.

The honest caveat is that this is a benchmark paper shipping with its own proposed method attached, and papers that do that carry a house-team advantage on the leaderboard they define. What the retrieved abstract does not give you is a scorecard by name for specific commercial or open systems, or a breakdown of audio-only quality metrics beyond LLM judging, so the 'state of the art still struggles' line reads as a category claim rather than a competitive ranking.

Where this points is the practical bit for anyone building on top of generative video: dubbing, ADR, lip-sync, sound design for auto-generated clips. These are the workflows where cross-modal editing quality decides whether the output is shippable at all, and a public yardstick makes it easier for smaller teams to pick a base model on capabilities the marketing decks tend to leave out.