Motion-Omni jointly generates speech and body motion, 5.4x faster

The default way to build a talking avatar runs the speech through one model and then hands the finished audio to a separate motion model. The Motion-Omni paper from Chengqian Ma and co-authors proposes collapsing the two into a single pass.

Their reported real-time factor is 0.78, faster than the audio it is generating, and 5.4 times the throughput of the same teacher cascade the system distills from. On motion metrics it "matches the same-audio teacher cascade to within 2% on reference-free motion metrics," according to the abstract, while beating non-teacher cascades on beat correlation and diversity. Word error rate sits at 2.62%, the lowest among the omni-modal systems the paper compares.

Joint training is doing the work. The authors are blunt about what happens if you shortcut it: "with the speech pathway frozen, motion remains misaligned with the audio, and co-adapting the LLM, Speech Generator and Motion Generator under both objectives is what recovers alignment while retaining spoken-dialogue ability." The backbone is a Qwen2.5-7B-Instruct model, and the training set is 422,856 quality-ranked pairs, or 1,402 hours, generated by pseudo-labeling speech responses with a replaceable motion teacher.

The release also includes SwDA-500 and what the authors describe as the "first public evaluation protocol for stochastic open-ended full-body spoken dialogue," bundling audio matching across motion systems, rendering, automatic metrics, human evaluation and latency measurement. Whether the two-percent gap and single-benchmark comparison hold up when other groups run the same protocol will decide how durable this result is.