On August 9, NVIDIA released NemotronLabs VoiceChat 11B, an open 11B end-to-end speech model that listens and speaks at the same time, with roughly 450 ms turn-taking latency and live tool calling, the first open full-duplex model of its kind.
What Was Released
NVIDIA published the weights and a container for NemotronLabs VoiceChat 11B on August 9, with the model card hosted on Hugging Face. The key architectural decision is that it is not a cascade: instead of chaining a speech recognizer, a language model, and a text-to-speech engine, the model performs streaming speech understanding and speech generation in one unified network. That removes the orchestration layer and the API handoffs a typical voice assistant stack needs between its components, which is where most of the latency in voice agents comes from.
The model is a hybrid Mamba/Transformer built from three existing NVIDIA components plus one new output path. A Fast Conformer speech encoder from Nemotron-Speech-Streaming-En-0.6b continuously encodes the incoming 16 kHz audio stream. The Nemotron Nano v2 LLM backbone consumes the audio tokens and predicts text tokens. An NVIDIA TTS decoder and codec renders the agent’s speech as 22.05 kHz audio. On top of that, a separate output channel is dedicated to tool-calling scripts, which is what makes live tool use during a conversation possible.
Training used roughly 550,000 hours of audio across real and synthetic corpora, building on NVIDIA’s earlier SALM-Duplex and Audio Flamingo 3 research lines. The result is a model that emits three streams at once: agent audio, agent text, and a running transcription of what the user said.
Full-Duplex, Not Turn-Taking
The defining behavior of the model is that it does not wait for the user to finish speaking. On the Full-Duplex-Bench 1.0 benchmark, NVIDIA measures smooth turn-taking at 448 ms latency, and the model’s take-over rate, the ability to yield when the user barges in mid-turn, is 1.00 at 480 ms. In plain terms: the user can interrupt while the agent is talking, and the agent steps aside cleanly.
That changes the feel of voice interaction compared with classic voice assistants, where each side waits for the other to stop. Barge-in capable agents are the practical benefit: drive-thru ordering, phone menus, and in-car assistants all suffer when a fixed “stop talking to process” pause sits between the user’s words and the system’s reply.
Tool Calling While Talking
The most novel part is live tool calling. VoiceChat 11B is the first open full-duplex model where tool calls happen while the conversation keeps flowing. Tool calls are emitted on the side channel as a TOOLCALL block, and application code returns results in a TOOL_RESPONSE block. Because tool calls in a cascade system would normally leave dead air while an API runs, NVIDIA added operator-defined “on-hold” lines: the moment the model generates the text that triggers a call, the agent speaks the configured line, so the user hears “let me check that for you” instead of silence.
The constraints are explicit and worth reading before you plan a deployment. NVIDIA recommends a maximum of five tools per session, the model cannot reliably call multiple tools simultaneously, and the user cannot interrupt the agent during tool execution. System prompts and tool responses must be ASCII-only and TTS-friendly. These are real limits, not documentation boilerplate: multi-tool orchestration is genuinely hard for an 11B model, and the on-hold line mechanism only works when each tool has its own canned phrase.
On the spoken tool-calling benchmark AU Harness BFCL-v3, the model scores 58.5 percent on simple calls, 62.5 percent on multiple, 42.5 percent on parallel, and 27.5 percent on parallel-multiple, with 89.6 percent on irrelevance rejection and a 56.1 percent average. On Full-Duplex-Bench v3 it reaches 82.5 percent tool selection and 44.2 percent argument accuracy. Those numbers say the tool calling works, but the parallel-multiple score of 27.5 percent is a warning that stacking tools in one utterance is not production-ready.
Deployment Reality Check
The licensing is permissive enough for pilots: the weights and container are public, and the model card carries the OpenMDW 1.1 research license. NVIDIA itself is careful with the framing, stating that the checkpoint is ready for research purposes only, and the model repository documents real failure modes. The two-minute audio context ceiling means long conversations need external memory management. The model can degrade into non-recoverable gibberish after several turns, which is the kind of failure that is survivable in a demo and fatal in customer service. There is also documented runaway self-talk after a turn ends, and dropped words in user transcription.
Hardware is the other gate. Running the model requires a single GPU with at least 80 GB of VRAM: an A100, H100, RTX 6000 Pro, or B200 on x86_64 Linux. That rules out consumer cards entirely. There is no hosted API and no inference provider currently serving the model, so teams without 80 GB GPU access cannot evaluate it at all.
NVIDIA reports the model ranks second among open full-duplex models on VoiceBench, which is a respectable position for a first release, but the failure modes and the hardware floor both point the same direction: this is a research milestone and a pilot candidate, not a drop-in product.
Who Should Care
The realistic buyers are teams that already own 80 GB GPUs and have a voice product in the pipeline: contact center and CX platforms, automotive in-cabin assistants, retail and drive-thru ordering, telecom IVR modernization, games with NPC dialogue, and accessibility tooling. The model removes the biggest cost of voice agents today, which is running three separate models and gluing them together, and it does so while keeping the conversation interruptible.
For teams building voice agents, the comparison that matters is with ByteDance’s SeedRealtime, which fuses audio, video, and text into a single full-duplex model and runs on a lighter footprint. We covered that release in our SeedRealtime review, and the two models frame the current state of open full-duplex speech: NVIDIA leans toward production speech quality with an 80 GB requirement, ByteDance leans toward multimodal breadth at lower cost. Neither is yet the easy default for a startup without serious GPU infrastructure.
The Honest Take
VoiceChat 11B is the strongest open full-duplex speech model NVIDIA has shipped, and the 448 ms turn-taking with a 1.00 take-over rate is genuinely good. The tool-calling side channel with on-hold lines is a clever answer to a real product problem, and the 550k hours of training data show up in the naturalness of the speech output. But the release is explicitly research-grade: 80 GB of VRAM, a two-minute context ceiling, documented degradation modes, and no hosted option. Anyone deploying it in front of real users is taking on the failure modes themselves.
The smart play is to treat it as the benchmark and the proof of concept: it shows what a unified full-duplex architecture can do, and it gives voice teams a reference point for measuring both NVIDIA’s next release and the competitors it has already forced to respond. If you have the GPU, run the model and time your own tool-calling flows, because the published benchmarks are on synthetic sets and your traffic pattern will differ. If you do not have the GPU, wait for the hosted version or for the smaller-footprint competition to close the gap, because the gap between a research release and a production voice product is still wide.