NVIDIA Opens NemotronLabs VoiceChat 11B With Tool Calling
TL;DR
- Artificial Analysis's two-axis benchmark (Full Duplex Bench plus Big Bench Audio) makes VoiceChat the sole open-weight model on the pareto frontier across both conversational dynamics and speech reasoning simultaneously.
- The proprietary-to-open gap on speech reasoning runs roughly 3:1 to 4:1, with Step-Audio R1.1 and GPT-Realtime at 87-96% versus VoiceChat's 29.2%.
- Tool-calling accuracy splits sharply: the model selects the right tool 82.5% of the time but fills arguments correctly only 44.2% of the time, making multi-step agentic voice tasks unreliable.
A model card that went up on Hugging Face today is worth a closer look, because it is the first open-weight full-duplex voice model to ship with live function calling. NVIDIA's NemotronLabs VoiceChat 11B is an end-to-end speech model that skips the usual ASR to LLM to TTS pipeline and does the whole loop in one hybrid Mamba/Transformer stack. It lands two days after NVIDIA open-sourced the NOOA agent framework.
The build is not exotic. NVIDIA takes its Nemotron Nano v2 9B language model, puts a Fast Conformer speech encoder in front, an NVIDIA TTS decoder behind, and adds a separate output channel that emits tool-call scripts while the audio conversation keeps going. The model card reports around 450 ms latency on smooth turn taking, 480 ms on user interruption, and a #2 ranking among open full-duplex models on VoiceBench. Tool calling scores 56.1 percent average on BFCL-v3, with 82.5 percent on tool selection but only 44.2 percent on argument accuracy and 33 percent Pass@1.
Even readers with no plans to build a voice agent should care, because the closest thing to this in production has been closed, hosted, and priced per minute. Having a reference full-duplex model with live tool calling that runs on A100, H100, H200, B100, B200 or RTX-6000 hardware via vLLM gives researchers and infrastructure teams something concrete to poke at, benchmark against, and copy architectural ideas from. The card explicitly positions it against PersonaPlex, Moshi and Freeze-Omni, none of which the card credits with tool calling.
Two things pull against the release: the license and the numbers. NVIDIA is shipping under its OpenMDW v1.1 license for research purposes only, so a startup can study the checkpoint but cannot ship a product on it. And the tool-calling scores, especially 33 percent Pass@1 and 42.5 percent on parallel tool calls, suggest this is a first-generation open attempt rather than a drop-in replacement for a hosted voice API. Missing from the card and early coverage: a real-world deployment cost, a third-party latency reproduction, and a comparison against closed hosted voice systems in blind tests.
The forward-looking read is that the interesting axis of competition in voice agents is moving from raw speech quality toward the mundane plumbing that makes agents useful, tool calls that fire mid-sentence without breaking the conversation. Once a reference open implementation of that plumbing exists, the closed voice APIs stop being the only place to see it working.
What others are reporting
-
Artificial Analysis Read →
Independent benchmark firm that originated the two-axis pareto framework cited in the primary alert; provides raw scores for all open-weight competitors and proprietary reference points.
Nemotron 3 VoiceChat (V1) is the only open weights model that performs amongst the top 3 on both dimensions.
-
NVIDIA Developer Blog Read →
First-party architecture walkthrough covering the Parakeet encoder plus Nemotron Nano v2 backbone plus TTS decoder stack, target industries, and sub-300ms latency design goal.
VoiceChat directly analyzes audio input and generates audio output in a unified and streaming LLM architecture.
-
NYU Shanghai RITS Read →
Academic technical review that foregrounds deployment constraints: 80GB VRAM minimum, two-minute audio context ceiling, no backchanneling support, and tool-argument accuracy well below tool-selection accuracy.
the only open weights model that performs amongst the top 3 on both — making it the clear leader on the pareto frontier.
Originally reported by huggingface.co
Read the original article →Original headline: NVIDIA Ships Open-Weight Nemotron VoiceChat 11B, First Open Full-Duplex Model With Live Tool Calling