NVIDIA Nemotron 3 Diarization Tops Voice Arena Benchmark with 14.72% DER

Speaker diarization classifies who spoke when, complementing automatic speech recognition, which transcribes what was said. A transcript from a meeting, customer call, or podcast may contain correct sentences but no speaker attribution, making it difficult to tell who made a commitment, raised an objection, or interrupted. Diarization identifies the time intervals during which each speaker is active, including intervals where people talk over one another. Those speaker timestamps can then be combined with ASR to create a speaker-attributed transcript.

Earlier models such as NVIDIA Streaming Sortformer established this approach for four-speaker diarization, including the streaming diar_streaming_sortformer_4spk-v2.1 checkpoint used as the baseline in the evaluations. Nemotron 3 Diarization expands support to eight speakers and improves the accuracy and throughput measured in those evaluations.

Diarization systems must solve two related problems. First, they must detect speech and assign it to the correct speaker. Second, they must preserve that assignment throughout the conversation, even after silence, interruptions, or long gaps between a speaker's turns. Streaming makes the second problem harder because a streaming system receives only a small chunk of new audio and limited context, unlike an offline model that can inspect an entire recording at once. Without an effective memory mechanism, the speaker assigned to one output channel in the current chunk can be assigned to a different channel in the next.

Nemotron 3 Diarization follows the Sortformer approach of ordering output speakers by when they first appear. The first new voice becomes the first speaker channel, the next new voice becomes the second, and so on. This arrival-time ordering makes the model's generic speaker labels stable and removes the need to solve a new speaker permutation for every chunk.

The model was trained using public and licensed speech data, including multispeaker-annotated, real-world conversations licensed from David AI. Additional licensed David AI audio provided source material for large-scale simulated English and multilingual mixtures spanning 21 languages. Adding David AI data to the training decreased compound diarization error rate by 0.77 absolute points, from 11.19% to 10.42%, at both the offline-style and ultra-low-latency operating points.

The model supports as many as eight speaker channels. These are anonymous labels, not real-world identities: the model can report that speaker_2 spoke from one timestamp to another, but it does not determine that speaker_2 is a particular person. Downstream applications can map these anonymous channel IDs to explicit speaker identities by pairing timestamps with meeting metadata, user profiles, or active speaker verification models.

The architecture accepts 16 kHz, single-channel audio and converts it into Mel-spectrogram features with a 10 ms frame step. It stacks those features by a factor of eight, producing 80 ms frames for a 31-layer Transformer encoder with rotary positional embeddings (RoPE). Above the Transformer, a Conv1D layer upsamples the predictions to the input-feature resolution. The default output is a [T, 8] floating-point tensor: T time steps by eight possible speaker channels. Each value is the probability that a speaker is active at that time. The default stride is 10 ms and can be configured to another multiple of 10 ms.

This representation handles overlap naturally. If two people speak at the same time, two channels can be active in the same frame. Postprocessing converts these probabilities into generic speaker labels with start and end timestamps. During streaming inference, two forms of memory provide context: the Arrival-Order Speaker Cache retains useful information about speakers observed in earlier chunks, organized according to their arrival-ordered channels, and a first-in, first-out queue supplies context.