← Research
Evaluation

DAI-ASR-I18N: Benchmarking Multilingual Conversational ASR

David AI’s DAI-ASR-I18N benchmark measures how well automatic speech recognition systems transcribe and diarize real, unscripted conversations.

Hours of conversation
147
Languages
21
Public eval set
~2hrs/language
Private eval set
~5hrs/language
Models evaluated
26systems

Results

Transcription Accuracy

Word / Character Error Rate (WER / CER) - lower is better

Dataset
Audio input
Metric
Model access
Loading results…
Speech recognition results for WER/CER, separate speaker channels. Scores by language. Unweighted mean of available per-language scores in this view; combined WER/CER views average different scoring units.

Diarization Accuracy

Overall Diarization Error Rate (DER), all overlap levels - lower is better (click each row to expand)

Dataset
Metric
Model access
Loading results…
Speaker diarization results for DER, combined audio. Scores by language. Published overall score pooled using the metric's reference units.
Changelog

v1.4 ()

  • Added ElevenLabs Scribe v3 and Reson8 Rumble-2 to the transcription and diarization leaderboards with results covering 21 and 20 supported languages, respectively.
  • Added Amazon Nova 2 Sonic, NVIDIA Parakeet TDT 0.6B v3, and IBM Granite Speech 5.0 2B to the transcription leaderboard with results covering 7, 7, and 6 supported languages, respectively.
  • Added Salesforce SOV-Transcribe-1 to the transcription leaderboard with results covering 6 supported languages.
  • Transcription results for newly added and re-scored systems use every clip in the evaluation set. A request that still fails after retries, including one that reaches the system's output limit, is scored as an empty transcript (100% error) rather than dropped.

v1.3 ()

  • Added NVIDIA Nemotron 3.5 Transcribe to the transcription and diarization leaderboards.
  • Added pyannote Precision-3 to the diarization leaderboard.
  • Added NVIDIA Nemotron 3.5 ASR to the transcription leaderboard. Results cover 14 supported languages.
  • Added Cohere Transcribe (03-2026) to the transcription leaderboard. Results cover 11 supported languages.

v1.2 ()

  • Added Voxtral Small Transcribe 3.0 and Voxtral Mini Transcribe 3.0 to the transcription and diarization leaderboards. Results cover the All, Public, and Private datasets.

v1.1.2 ()

  • Added Mandarin results for Grok Voice Transcribe 2.0 and updated its aggregate scores across the All, Public, and Private datasets.

v1.1 ()

  • Updated the methodology of diarization benchmark in diarization error rate (DER).
    • We applied the following post-processing steps to the model predictions to report the diarization metric.
      • Segment-level timestamps: we do not perform additional post-processing to contiguous segments with timestamps of activated speech regions. This includes models that are specific to diarization task (e.g. Nemotron 3 Diarization) and transcription models that output contiguous segment timestamps (e.g. gpt-4o-transcribe-diarize)
      • Word-level timestamps: consecutive words from the same speaker separated by gaps of up to 200 ms are merged into a single segment, matching the post-processing of the ground-truth references. This includes Grok Transcribe 2.0, Scribe V2, MAI-Transcribe 2, Gemini 3.5 Transcribe.