Modus Puts 15 Modalities in One Decoder-Only Any-to-Any Model

Any-to-any multimodal generation has usually meant stacking specialist heads and task-specific losses on top of a base model. A new paper on Hugging Face titled Modus: Decoder-Only Any-to-Any Modeling of Diverse Modalities argues you can drop the scaffolding. The team extends the pretrained BAGEL backbone into a decoder-only Transformer that treats fifteen modalities symmetrically, from RGB and depth to canny edges, SAM masks, DINOv2 features, CLIP embeddings, visual grounding boxes and object detection, with a project page at modus-multimodal.epfl.ch.

The architecture is a Mixture-of-Transformers with two experts, a 1D expert doing next-token prediction on discrete sequences and a 2D expert doing flow matching in latent space for spatial modalities. A key finding is that the default logit-normal timestep schedule caused modality confusion, for example the model producing surface normals when depth was requested. Uniform timestep sampling fixed it and pushed GenEval from 0.77 to 0.81 in ablations. Training ran in three stages across 30B, 20B and 15B tokens for roughly 5,664 GH200 GPU-hours, and two checkpoints are released, Modus-15modality-14B-A7B and Modus-15modality-77B-A13B, alongside a 29M-sample dataset built by annotating BLIP-3o images with DepthAnything, Marigold, Grounded-SAM, GLaMM and ViTDet.

What is genuinely new is the emergent behavior that falls out of a unified decoder. Because generated outputs can be fed back as conditioning, Modus supports chained generation, and on NYUv2 the RGB to canny to surface normal chain scored 19.87 versus a 20.02 direct baseline. It also supports cross-modal self-verification: scoring generated images with the same model's grounding and VQA outputs lifted GenEval from 0.81 to 0.84 without an external verifier.

The honest caveat is that Modus does not top BAGEL on every task. MMMU sits at 51.1 against BAGEL's 53.2 and text-to-image GenEval is 0.81 against 0.86, and coverage is vision-heavy with audio and 3D flagged as future work. What the paper does not give you is inference latency for chained pipelines on real hardware, or the licensing terms attached to the released checkpoints and dataset.

For anyone building perception stacks, the payoff is fewer specialist models to train, host and version, plus a natural way to sanity-check outputs by asking the same model a different question. Whether the 77B checkpoint proves practical outside a research cluster is the next thing worth watching.