NVIDIA's Nemotron-Audex unifies audio and text in one decoder

A quiet trend in multimodal work has been that bolting audio or speech onto a strong text model usually costs you something on text. NVIDIA's new paper on Hugging Face makes the opposite argument, describing Nemotron-Labs-Audex-30B-A3B as a unified audio-text LLM that handles audio understanding, speech recognition and translation, TTS, and speech-to-speech generation through a single transformer decoder while keeping text reasoning essentially intact.

The design starts from Nemotron-Cascade-2-30B-A3B, a text-only MoE LLM, and treats both modalities uniformly inside one decoder. Audio inputs are encoded and projected into the text embedding space; on the output side, text tokens and quantized audio tokens are generated the same way. The training run used 157.4B audio tokens and 320.5B text tokens, combined with multi-stage supervised training, text-only Cascade RL, and multi-domain on-policy distillation. A 2B variant ships alongside the 30B-A3B model, and both checkpoints are on Hugging Face for open research.

If the results hold up under independent testing, the interesting part is not any single benchmark headline but the claim that unification does not have to be a tax on text. For teams building assistants, call-center copilots, or voice-first products, that would collapse a stack that today usually means a text LLM plus a separate ASR model plus a separate TTS pipeline into one model with one set of infrastructure. Open weights also mean groups that cannot afford closed voice APIs get a real starting point rather than a demo.

The honest caveat is that this is a paper from the vendor whose model it evaluates, and its own framing is careful — it talks about strong text reasoning with marginal or no regression rather than a clean head-to-head win on every text benchmark against the base model. What the writeup does not give you is inference cost, latency on the audio-out path, or behavior on non-English and noisy real-world speech. Those are the numbers that will decide whether Audex becomes a default backbone for voice apps or stays a research artifact.