SJTU, Ant Group Survey Charts Three Paths to Omnimodal Embeddings

Embedding research across text, code, image and video, audio, graph, table and time series is quietly converging on a shared recipe even while the publishing communities stay walled off, a team from Shanghai Jiao Tong, Australian National University, Tongji, NUS and Ant Group argues in a position paper posted to Hugging Face on November 3. A tally in the appendix sharpens the gap: of 1,352 formally published embedding papers from 2023 to 2025, plus a decade of arXiv preprints, "works that explicitly cross modality boundaries account for under 5% of both samples and emerge only recently."

The authors anatomize every modern embedder into five components — tokenization, encoder, aggregation and representational object, training objective, and evaluation paradigm — and then propose a 3C lens on top. Coverage is "which inputs the system has actually encoded and tested." Comparability is whether separately encoded inputs can be matched for a specific task. Compositionality, the hardest, is "whether an output built from several inputs genuinely depends on all of them." Three paths map onto those capabilities: "transformation broadens coverage, alignment enables comparability, and joint embedding, the terminus we advocate, aims at compositionality," the paper argues — a framing that puts today's CLIP-style setups at the alignment rung, not the final one.

The convergence they point to rests on four pillars: contrastive learning as a universal objective, Transformers as universal encoders, a pretrain-then-finetune recipe, and a shift from token- or pixel-level objectives to sequence- or image-level ones. This is a survey, not a benchmark, so no joint-embedding model is held up as satisfying all three Cs today, and the central claim — that each modality must borrow the others' hard-won techniques — is advocacy, not measurement. It lands into a busy stretch for the subfield; our multimodal tracker has logged 102 stories in the last 90 days.