Omni-Encoder packs image, video, audio into one SigLIP-2 ViT

Omni-Encoder runs image, video, audio, and audio-visual inputs through a single SigLIP-2 Vision Transformer with fully shared parameters, instead of stitching together separate encoders. On arXiv, the authors argue joint end-to-end training beats the dominant pattern of 'post-hoc binding to a frozen anchor space.'

The design has three moving parts. A '1D-convolutional audio tokenizer' converts mel-spectrograms into tokens aligned with visual patches. A rotary position embedding parameterized by absolute timestamps places video frames and audio segments in a unified spatiotemporal coordinate system. Training then 'interleaves image, video, audio, and temporally aligned audio-visual batches and accumulates their gradients into one update,' which the paper calls 'central to stable joint optimization.'

The headline claim: Omni-Encoder 'achieves competitive results on 16 benchmarks across all four modalities with one shared parameter set,' and 'joint training improves each modality over its separately trained counterpart.'

The abstract names none of the 16 benchmarks, publishes no per-modality numbers, and names no comparator model. Code and models are promised but not yet public.