Changelog
v1.4 ()
- Added ElevenLabs Scribe v3 and Reson8 Rumble-2 to the transcription and diarization leaderboards with results covering 21 and 20 supported languages, respectively.
- Added Amazon Nova 2 Sonic, NVIDIA Parakeet TDT 0.6B v3, and IBM Granite Speech 5.0 2B to the transcription leaderboard with results covering 7, 7, and 6 supported languages, respectively.
- Added Salesforce SOV-Transcribe-1 to the transcription leaderboard with results covering 6 supported languages.
- Transcription results for newly added and re-scored systems use every clip in the evaluation set. A request that still fails after retries, including one that reaches the system's output limit, is scored as an empty transcript (100% error) rather than dropped.
v1.3 ()
- Added NVIDIA Nemotron 3.5 Transcribe to the transcription and diarization leaderboards.
- Added pyannote Precision-3 to the diarization leaderboard.
- Added NVIDIA Nemotron 3.5 ASR to the transcription leaderboard. Results cover 14 supported languages.
- Added Cohere Transcribe (03-2026) to the transcription leaderboard. Results cover 11 supported languages.
v1.2 ()
- Added Voxtral Small Transcribe 3.0 and Voxtral Mini Transcribe 3.0 to the transcription and diarization leaderboards. Results cover the All, Public, and Private datasets.
v1.1.2 ()
- Added Mandarin results for Grok Voice Transcribe 2.0 and updated its aggregate scores across the All, Public, and Private datasets.
v1.1 ()
- Updated the methodology of diarization benchmark in diarization error rate (DER).
- We applied the following post-processing steps to the model predictions to report the diarization metric.
- Segment-level timestamps: we do not perform additional post-processing to contiguous segments with timestamps of activated speech regions. This includes models that are specific to diarization task (e.g. Nemotron 3 Diarization) and transcription models that output contiguous segment timestamps (e.g. gpt-4o-transcribe-diarize)
- Word-level timestamps: consecutive words from the same speaker separated by gaps of up to 200 ms are merged into a single segment, matching the post-processing of the ground-truth references. This includes Grok Transcribe 2.0, Scribe V2, MAI-Transcribe 2, Gemini 3.5 Transcribe.
- We applied the following post-processing steps to the model predictions to report the diarization metric.