NVIDIA Nemotron 3.5: Leading Open-Weight Model for Intelligent Multilingual Transcription

Nemotron 3.5 Transcribe, developed by NVIDIA, represents a significant leap in speech processing technology, moving far beyond simple word-for-word transcription. It is an open-weight, highly adaptable, and promptable model designed to transform raw recorded audio into structured, actionable, and contextually rich text. The model’s core strength lies in its ability to handle the complexities of real-world speech, such as multilingual conversations, fast-paced meetings, and specialized medical dictations, which often require more than just a transcript—they require speaker identification, accurate translation, and domain-specific formatting.

Initially, a basic transcript only captures the spoken words, leaving critical questions unanswered: Who said what? How should this be translated? And how should domain-specific jargon be formatted? Nemotron 3.5 Transcribe addresses these gaps by integrating multiple advanced functionalities into a single, unified checkpoint. This comprehensive package includes robust transcription across 40 languages, speaker attribution (also known as diarization) for up to 13 languages, translation capabilities for 31 languages to or from English, and even the ability to answer questions based on the audio content. Furthermore, it is engineered to process recordings up to an hour in length and can accurately label up to eight distinct speakers, making it suitable for complex, multi-person interactions.

One of the most revolutionary aspects of the model is its level of control and adaptability. Developers are not limited to a single output format; they can guide the model's output using natural language prompts. This means the same audio recording can be processed multiple ways simply by changing the instruction. For instance, a developer can prompt it to simply 'Transcribe this recording and label each speaker,' or they can ask a higher-level question like, 'What decisions were made in this meeting?' This capability allows the model to function not just as a transcriber, but as an intelligent meeting analyst. The model’s ability to handle challenging audio, such as the rapid-fire, heavily overlapping dialogue characteristic of classic films like His Girl Friday (1940), demonstrates its resilience and advanced understanding of speech patterns.

Beyond general transcription, the system excels in domain-specific applications, particularly in healthcare. In a medical context, the model allows for proactive context definition before training. Developers can provide a list of possible terms (e.g., HbA1c, metformin) and instruct the model to use standard medical spelling and write numbers in conventional written form, ensuring the output meets strict professional standards. This level of control is achieved through the integration of Granary v2, which provides multilingual data and a reusable curation pipeline. Coupled with NeMo Speech fine-tuning tools, this allows developers to adapt the model to the specific speech patterns and vocabulary encountered in their applications. This fine-tuning capability is highly impactful, as demonstrated by a medical case study where fine-tuning cut the Word Error Rate (WER) on held-out speakers by an impressive 45.8% compared to the best prompt-based performance.

Performance benchmarks solidify Nemotron 3.5 Transcribe's position as a market leader among open-weight models. On the Hugging Face Open ASR Leaderboard, it ranks first among open-weight models for English (5.33% average WER) and maintains the lowest mean WER across seven languages in the multilingual track (4.62%). Its performance on long-form audio is also top-tier (10.98% mean WER). Furthermore, its technical efficiency is remarkable. It ties for first among open-weight models on Artificial Analysis’s AA-WER leaderboard at 2.8% AA-WER. Crucially, this high accuracy is achieved while utilizing only about 4 billion active parameters out of a total of 30 billion parameters. This efficiency places it on the open-weight Pareto frontier of accuracy versus active parameters, meaning it achieves state-of-the-art performance with significantly fewer computational resources than many proprietary or larger models. The model achieves this efficiency by pairing its speech encoder with a hybrid Mamba–Transformer mixture-of-experts language model, ensuring that only a small, highly optimized subset of parameters is active for each token processed.

In practical deployment, the model’s versatility is evident. For educational settings, platforms like Agora utilized word-level timestamps to transform a recorded class into a dual-view transcript, allowing students to see exactly what was said and precisely when it was said, enhancing the learning experience. The model also supports word-level timestamps, which is vital for applications requiring precise synchronization between audio and text, such as subtitling or educational replay tools. While the model card provides comprehensive guidance, including task-by-language matrices and recommended presets, users are advised to always spot-check critical details—such as names, speaker changes, and answers—against the original recording, as is standard practice with any advanced speech AI.