DAI-S2S-ST: Evaluating S2S Model Quality with Human Preference Data

Overview

The David AI S2S Single Turn (DAI-S2S-ST) leaderboard is a benchmark that evaluates S2S model quality on single-turn human-recorded audio prompts focused on real consumer use cases. Quality is evaluated by human comparative preference ratings across 12 dimensions of quality (11 specific dimensions + 1 “overall” rating), which are grouped into “Humanness” and “Technical Quality” categories for analysis. Comparative ratings are used to fit a Bradley-Terry Elo model, and confidence intervals are reported for each Elo score.

Two voices and two thinking levels (where supported) were included for each evaluated model (except the cascade baseline). These voices and thinking levels are aggregated into a single model-level Elo on the leaderboard; the voice and thinking-level comparisons are reported as their own figures in the results section. The audio prompts themselves are kept private to mitigate eval set leakage risk - the audio samples included in the blog post are representative, but not used in the leaderboard itself.

Audio prompt generation

Audio prompt scripts are drawn from eleven Objective Categories and carry a difficulty level from L1 (easiest) to L5 (hardest), plus a separate prompt set captured under degraded acoustic conditions. Each script is then recorded by a human contributor on the David AI audio data collection platform. The samples used in the leaderboard consist of ~14 input prompt scripts for each difficulty level x objective category pair for a total of 819 audio prompts (11 objective categories x 5 difficulty levels x ~14 prompts + ~45 degraded audio samples).

The objective categories and audio samples are described in the main blog post.

Each script was sent to David AI voice contributors to be recorded on our audio platform. Each script was recorded by at least one contributor in the target acoustic environment: clean, meaning minimal background noise and a near-field microphone setup; or degraded, meaning indoor background noise, outdoor background noise, or a far-field microphone setup. The recorded samples were passed through an automated and human review QA process, and one audio sample was selected for each unique script. The selected samples were volume normalized and trimmed, but otherwise unprocessed.

Model selection

We surveyed the available S2S leaderboards in July 2026 and identified what we believed to be the strongest S2S model offered by each of the SOTA S2S model providers available at the time. For model families that supported multiple thinking modes, we selected the highest and lowest available variants for comparison. In practice, our analysis shows negligible performance differences on our eval set for different thinking levels, and our results are aggregated across thinking levels in our leaderboard for simplicity (the comparison by thinking level is reported in the results section).

In addition, we identified a SOTA cascade model pipeline and a high-performing open-source model to serve as comparison baselines. The cascade pipeline uses Scribe-V2 for speech-to-text (STT) transcription, GPT-5.6 Terra as base language model, and ElevenLabs Flash v2.5 for text-to-speech (TTS). These models were chosen as a balance of quality and latency, as the highest audio quality TTS models have high latency and are not a realistic comparison point for deployable S2S systems.

Finally, we included GPT-Realtime-2 (xhigh thinking) in addition to the GPT-Realtime-2.1 variants because GPT-Realtime-2.1 was framed by the developers as an incremental update with specific feature support and latency optimization; we considered it plausible that GPT-Realtime-2 might outperform 2.1 on human preference over our eval set once latency differences were removed. In practice we see only a minimal performance difference across GPT-Realtime-2 and GPT-Realtime-2.1.

The final list of models evaluated at launch time includes:

We expect to add more models and variants to this leaderboard over time. The full list of 19 model configurations is available in the Model Configuration Details section.

Voice selection

We selected two voices (one male, one female) for each S2S model except the cascade baseline. The data for each voice is aggregated in our leaderboard for simplicity; the comparison between a model’s two voices is reported in the results section.

To minimize the impact of voice preference on the ranking, we prioritized selecting voices that are:

• Neutral EN-US accented
• Medium pitch range within each gender (eliminating any voice with a noticeably high or low pitch)
• No strong sociolinguistic markers (eliminating any voice that is noticeably or characteristically from a particular sociological group)

When a model had a default voice identified in the documentation that fit the above criteria, we used that voice. When the default voice did not fit the criteria, or when there was no default voice specified, we selected a random voice that met the criteria.

The model that was most impacted by this methodology was Grok voice-think-fast 2.0, which uses a default voice with an EN-GB accent (Eve) that we replaced with another voice (Ara).

The final voices used for each model as of launch time were:

Model Inference

Each combination of <model, thinking level, voice> was prompted with each of the 819 audio input prompts using their officially recommended inference settings to the extent possible.

We did not hand-tune system prompts, user instruction, or any other parameter settings for individual models. For models that provide a default prompt or other recommended settings in their public documentation, we used these default and recommended values - see the Qwen case below. For models that did not provide a default prompt or recommended settings, we used a standard instruction of “You are a helpful voice assistant” and used the default parameter values. The full set of model inference configuration values can be found in the Model Configuration Details section.

The model configuration that stands out as the most unusual is the system prompt used for the Qwen3 Omni model:

You are a virtual voice assistant with no gender or age. You are communicating with the user. In user messages, "I/me/my/we/our" refer to the user and "you/your" refer to the assistant. In your replies, address the user as "you/your" and yourself as "I/me/my"; never mirror the user's pronouns—always shift perspective. Keep original pronouns only in direct quotes; if a reference is unclear, ask a brief clarifying question. Interact with users using short (no more than 50 words), brief, straightforward language, maintaining a natural tone. Never use formal phrasing, mechanical expressions, bullet points, overly structured language. Your output must consist only of the spoken content you want the user to hear. Do not include any descriptions of actions, emotions, sounds, or voice changes. Do not use asterisks, brackets, parentheses, or any other symbols to indicate tone or actions. You must answer users' audio or text questions, do not directly describe the video content. You should communicate in the same language strictly as the user unless they request otherwise. When you are uncertain (e.g., you can't see/hear clearly, don't understand, or the user makes a comment rather than asking a question), use appropriate questions to guide the user to continue the conversation. Keep replies concise and conversational, as if talking face-to-face

We initially implemented Qwen3 with the same “You are a helpful voice assistant” system prompt used for most other models, but we saw a high rate of catastrophic failure where the model would respond with incoherent content. After investigation, the Qwen3 Omni GitHub page includes a recommended prompt for “Audio-Visual interaction... where the input consists of a video and its corresponding audio.” While video is not provided in our audio input prompts, we found model responses much more stable using this prompt, and used this to generate the actual evaluation samples.

Audio requests were streamed to standard model inference API endpoints, and the audio responses were collected after the end of turn event was emitted by the model API. For models that provide parallel output streams to the audio, everything but the final audio output was ignored for purposes of rating. Specifically:

• Some models provide a “provisional” audio stream that outputs audio chunks as they become available while the final audio stream is still being constructed.
• Some models provide a separate “thinking” token stream.
• Some models provide an input and/or output transcript in addition to the output audio stream.

All of these values were recorded for potential future analysis, but ignored for the purposes of human evaluation.

Human Preference Rating Collection

A rating pair consists of the output from one specific model configuration <model, thinking level, voice> paired with the output from another configuration on the same input prompt. Rating pairs can include output from the same base model at different thinking levels and/or using different voices. A given model configuration is not compared against all other model configurations for every single input prompt; however, every model configuration had at least partial coverage over other configurations for every input prompt, and every model configuration was paired with every other model configuration at roughly equal rates across all rating pairs. At launch, 19 unique model configurations were evaluated.

For each rating pair, one or more raters were assigned to answer a set of up to 12 distinct questions (or “rubrics”), one of which was “overall preference”. The median number of raters for a given rating pair was 2 (i.e., the typical rating pair was rated by 2 different humans). 3 of the 12 rubrics are “gated,” meaning the question is only asked when it is applicable to the specific input prompt (for example, the “reasoning” question was only asked for input prompts tagged as requiring some amount of reasoning in the response). The full list of rubric questions is detailed in the main blog post.

For CMOS ratings, a single rater was asked to answer all relevant rubric questions for a given rating pair in succession. During collection, questions were grouped into three categories, and each category was answered in full before moving on to the next category:

These category groupings were reevaluated in the analysis stage, and statistical clustering regrouped them into the Humanness and Technical Quality categories as discussed in the main blog post.

MOS ratings were collected in addition to CMOS ratings for internal analysis, but the leaderboard reports only CMOS results.

Model Clustering Methodology

To group the rubric dimensions into categories, we cluster rubrics by the similarity of their per-model performance profiles. For each rubric, we take the per-model latent strengths from the same ordinal Bradley-Terry fit used for the leaderboard, yielding a rubric × model matrix. Two corrections are applied before clustering:

1. Because every rubric's scores are dominated by a shared "general quality" axis (the halo effect), naive clustering would group rubrics by how strongly they track overall quality rather than by what they measure; we therefore project out the first principal component of the (row-centered) matrix and cluster on each rubric's residual profile.
2. Because rubric pairs are scored by the same raters on the same prompts, their observed similarity includes a shared-noise component; we estimate this via a rater-and-prompt clustered bootstrap (1,000 resamples) and subtract it from each pairwise similarity. We then apply average-linkage hierarchical clustering to the corrected similarities.

We assess cluster reliability as the fraction of bootstrap resamples that reproduce the identical partition, against a chance baseline computed on signal-removed data. The two-cluster split of Humanness / Technical Quality reproduces in 82% of resamples (chance rate < 1%); a three-cluster split, which would separate the acoustic rubrics into their own group, reproduces in only 38% and is not reported.

Clustering was computed with GPT-Realtime-2 and 2.1 pooled as a single family (six model families); given their near-identical rubric profiles we do not expect the split to alter the structure.

As a held-out check, we also clustered two disjoint random halves of the rater pool independently: the two-cluster structure and the Humanness core are consistently recovered (mean Adjusted Rand Index 0.60 over 500 splits), though boundary rubrics (instruction following, perceived audio quality) occasionally migrate between clusters. We therefore report the k=2 grouping and treat finer sub-structure as provisional pending more data.

Model Ranking Methodology

The leaderboard uses CMOS preference scores to directly compute Elo from a Bradley-Terry model. For the primary leaderboard, we use the CMOS scores from the Overall question per model; for fine-grained analysis, we compute individual rankings across rubric categories, as well as per-voice and per-thinking-level variants.

Since CMOS score is an ordinal scale from [-3, 3], we use ordinal Bradley-Terry as the primary model to compute the latent score for each candidate. We also fit two other ranking engines: Massey ranking and classic Bradley-Terry with partial-margin extension (-3..+3 CMOS values normalized to -1..+1 and treated as partial win margin) for cross-method validation. All three engines produce near-identical rankings with each other (Spearman correlation at or above 0.99).

The latent Elo score for each candidate is anchored at 1000 for ease of reading, with a 400-point gap corresponding to 10-to-1 odds:

Elo = 1000 + (400 / ln 10) · s

Confidence intervals are calculated using rater-prompt-clustered bootstrapping: we randomly sample raters and prompts with replacement to build a subset and calculate ranking statistics. We repeat the process 1,000 times and report the 95% CI as the range between the 2.5th and 97.5th percentiles of the spread.