NVIDIA Releases Nemotron 3 Diarization, an Open-Weight Model That Tracks Up to 8 Speakers in Real Time

The model runs on Linux through NVIDIA NeMo and supports Ampere, Ada Lovelace, Hopper and Blackwell GPUs. It is aimed at a gap that automatic speech recognition leaves open: ASR returns the words but does not say who said them, so without attribution a summarizer cannot tell who made a commitment or who raised an objection. Diarization outputs the time intervals in which each speaker is active, and those timestamps combine with ASR output to produce a speaker-attributed transcript. Meeting tools, call analytics, podcast pipelines and voice-agent memory all depend on that step.

NVIDIA's earlier Streaming Sortformer checkpoint, diar_streaming_sortformer_4spk-v2.1, supported four speakers. Nemotron 3 Diarization doubles that limit to eight, and NVIDIA says the target is messy multi-party audio where people talk at once.

The model accepts 16 kHz single-channel audio in .wav, .flac, .opus or .mp3 format and converts it into Mel-spectrogram features with a 10 ms step. Those features are stacked by a factor of eight to produce 80 ms encoder frames, which are processed by a 31-layer Transformer encoder with rotary positional embeddings. A Conv1D layer upsamples the predictions back to 10 ms resolution, and the output is a [T, 8] tensor of per-speaker activity probabilities. Overlap is handled directly: if two people talk at the same time, two channels activate in the same frame. Speakers are ordered by arrival time under the Sortformer approach, so the first new voice takes channel 1 and the next takes channel 2, keeping labels stable across streaming chunks without re-matching speakers to channels for every chunk. Streaming relies on two memory mechanisms, an Arrival-Order Speaker Cache that keeps speaker information from earlier chunks and a FIFO queue that supplies recent frame context. The labels are anonymous, and mapping them to real identities is left to downstream applications.

Input-buffer latency equals the chunk plus right context multiplied by 80 ms, and NVIDIA lists four operating points. An offline-style configuration uses a 30.4-second buffer and reaches 12.73 percent DER on the DIHARD III full set at 15,113 times real-time factor with batch 32 and compiled code. Low latency uses a 1.04-second buffer at 13.18 percent DER and 865 times real time, very low latency a 0.64-second buffer at 13.28 percent and 579 times, and ultra-low latency a 0.32-second buffer at 13.55 percent and 292 times. Those figures exclude compute, networking and ASR time. The model can technically run with an 80 ms buffer, but 0.32 seconds is the lowest recommended setting.

In Voice Arena's initial Diarization-Bench results, the model ranked first among 12 systems and 17 configurations across 139 English conversations totaling about 22 hours. It scored 14.72 percent DER against 19.3 percent for the next-ranked system, roughly a 24 percent relative reduction. NVIDIA notes that these results may change once Voice Arena completes its Version 1 evaluation. Against the four-speaker baseline at 1.04 seconds of latency, DER dropped on all eight evaluation conditions, with relative reductions ranging from 9.0 percent on CALLHOME-Part2 to 65.2 percent on NOTSOFAR1 MHM and an unweighted mean of 41.0 percent across the eight conditions. There is one regression: on two-speaker CALLHOME at 30.4 seconds, DER rose from 5.68 percent to 5.98 percent. Full-set CALLHOME-Part2 still improved from 10.32 percent to 9.10 percent. Throughput also increased, reaching 15,113 times real time at 30.4 seconds against 2,619 times for the baseline. The tests used BF16 on an NVIDIA RTX PRO 5000 with torch.compile(), and the figures are batched numbers rather than single-stream application latency.

Training combined about 10,000 hours of real conversations with 82,611 hours of simulated multi-talker mixtures. The mix included real-world multi-speaker audio licensed from David AI, and adding that data cut compound DER from 11.19 percent to 10.42 percent. The licensed source audio for the simulated mixtures spans 21 languages.

Developers install NVIDIA NeMo Speech with Python 3.12 or later, then load SortformerEncLabelModel.from_pretrained with the checkpoint nvidia/Nemotron-3-Diarization and call diarize on audio files. Segments are returned as start, end and speaker_id, and pairing the model with Parakeet TDT 0.6B v3 through the ASR integration guide adds the words. A live demo Space offers synthetic conversations, a live microphone, a multilingual live microphone and audio upload.