DIAL: Dialogue Interaction Assessment Leaderboard

Overview

DIAL measures how well speech-to-speech (S2S) models hold live, multi-turn spoken conversations with people. To do so, we conduct and evaluate 9,055 conversations across three scenario types: Consumer, Business, and Proactive. A rating is one score on one dimension from our evaluation rubric. Some raters participate in more than one scenario type, so the per-type rater counts sum to more than the total number of raters.

**Table 1. Scenario types: what each measures, and the ratings, conversations, raters, and scenarios behind it.** Counts follow the latest methodology release scope. Total raters are deduplicated across scenario types.
Scenario type	Description	Measured by	Ratings	Conversations	Raters	Scenarios
Consumer	The rater chooses an everyday scenario that fits their own life and brings real personal details to it. There is no knowledge base or correct answer; quality is judged against the rater’s own goal.	Rater scores the model from 1-5 in four dimensions. Rater tags 24 defect options across four categories.	17,228	4,307	896	1,334
Business	The model is a voice agent for a company. Quality depends on staying within the company’s policies and knowledge base, in addition to how useful and natural it is.	Rater scores the model from 1-5 in four dimensions. Rater tags 24 defect options across four categories. Groundedness evaluated using LLM-as-a-judge, where a large language model (LLM) grades the transcript.	13,232	3,308	807	468
Proactive	The model phones a business to get a task done for a person. Quality depends on whether the task is done correctly.	Rater scores the model from 1-5 in five dimensions. Task success measured against expected outcome.	7,200	1,440	134	30
Total			37,660	9,055	1,093	1,832

Scenario types

Consumer

Background

Consumers use voice assistants for everyday tasks such as planning, learning, troubleshooting, and deciding. Quality depends on whether the model tracks context over the course of the conversation, adapts as the person’s intent changes, and keeps the conversation natural. The Consumer scenario type represents this broad range of intents.

Rater context

In these conversations, the rater sees the scenario title, a goal, and four prompts to think about before the conversation. They are encouraged to bring their real-life experiences into the conversation. Throughout the dialogue, they react to what the model says, push back, interrupt, and backchannel as a consumer would in a natural conversation.

For example, in the scenario “Evaluating job offer benefits and risks,” the rater sees this brief:

Goal: Analyze the benefits and risks associated with a job offer you're considering, in order to form a solid decision about whether to accept it or negotiate further.

What you should do:

- What specific benefits or downsides are you most concerned about in this job offer?
- Can you recall a time when you had to weigh the pros and cons of a job offer? What was the situation?
- If you were to receive this job offer right now, what factors would you prioritize in your decision-making?
- What are the key facts you need to gather about this job offer before making a decision?

Model context

In these scenarios, the model is given a default system prompt: “You are a helpful voice assistant.” This is the base instruction for every model in every scenario type.

Scenario diversity

1,334 distinct scenarios appear in the dataset across most combinations of six objectives and 10 domains. Some objective / domain pairs have no natural scenarios, so we exclude these (e.g., Coaching & Interpersonal Navigation in Tech and devices). To ensure diversity, we cap the number of individual raters per scenario and model.

**Table 2. Consumer objectives.** Objective definitions describe the scenarios included in this scenario type.
Objective	Description
Brainstorming & Ideation	Generate and develop options with a person who has a goal but no fixed answer in mind.
Coaching & Interpersonal Navigation	Work through a personal or interpersonal situation. The useful response is judgment and perspective, not information.
Decision & Recommendation	Narrow a set of options to a recommendation against the person’s stated constraints.
How-To Walkthrough	Walk through a procedure the person intends to carry out themselves.
Information & Explanation	Explain a concept, fact, or mechanism the person does not yet understand.
Learning & Practice	Build or rehearse a skill over the conversation, including correction and repetition.

**Table 3. Consumer scenarios, by domain and objective.** Distinct scenarios with included conversations. Scenario copies across batches count once.
Domain	Brainstorming & Ideation	Coaching & Interpersonal Navigation	Decision & Recommendation	How-To Walkthrough	Information & Explanation	Learning & Practice	Total
Commerce	6	0	44	0	0	0	50
Entertainment	27	0	93	13	40	25	198
Health and wellbeing	0	47	47	25	48	32	199
Home and everyday tasks	28	0	69	42	37	16	192
Personal life	17	31	45	26	1	29	149
Relationships	0	117	0	0	0	0	117
Social and general	0	25	0	0	0	0	25
Tech and devices	0	0	31	15	38	7	91
Travel and local	13	11	64	16	46	17	167
Work and business	13	48	56	4	13	12	146
Total	104	279	449	141	223	138	1,334

Evaluation

Conversations are evaluated on the mean opinion score (MOS) rubric and defect categories in the appendix.

Business

Background

Companies use voice agents for customer service. In these cases, the model must follow company policy, answer from a defined knowledge base, and stay accurate when the customer pushes back. Wrong information, a policy violation, or an unresolved request is costly. The Business scenario type tests whether a model is preferred by raters and grounded in the company knowledge/policies. It does not test tool use; the model has no access to live systems (e.g., looking up a booking or issuing a refund), and must answer entirely from the knowledge base provided in its context.

Rater context

In these scenarios, the rater plays a customer calling a fictional company about a specific account, product, or case. They see the situation, their goals for the conversation, their persona’s account details, and further information to share only when it becomes relevant. For example:

You've opened your account to find a string of fees charged on the same day and you can't make sense of them.

Your goals:

- Find out why you were charged four times in one day when you thought there was a cap
- Find out which of the four are overdraft fees and which are returned-item fees
- Find out why your linked savings didn't cover any of it
- Find out whether any of them can come off

Personal information: Harborlight Bank, existing customer since 2022-11-05, checking and savings accounts, card active, no fraud or dispute flag.

Share when relevant: four separate $35 fees on one day; two were card purchases, one was the rent check and one was an automatic utility payment; savings is linked for overdraft protection but nothing transferred; opted in to overdraft coverage for debit card transactions.

Model context

In these scenarios, the model is given the same base instruction as the Consumer scenarios, followed by a scenario-specific prompt with the company persona, operating rules, and knowledge base. For example, in the Harborlight Bank scenario, the model acts as the bank’s voice assistant with fee and overdraft policies in its context. Every model gets the same prompt for a given scenario. An example prompt can be found in the appendix.

Scenario diversity

The Business scenario type has 468 distinct scenarios across three objectives and four domains: Airline, Banking, Health insurance, and Telecom. Each domain is built around a fictional company, with a knowledge base and policies the model must follow. Each scenario also has one of five difficulty levels. At L1 and L2, the caller is cooperative, and the conversation stays on one topic. L3 to L5 add pressure from the caller, missing information, or several goals in one conversation.

**Table 4. Business objectives.** Objective definitions describe the scenarios included in this scenario type.
Objective	Description
Troubleshooting & Diagnosis	Something is not working or gives an unexpected result, and the conversation works toward a cause.
Information & Explanation	The caller wants to understand something: a policy, cost, rule, timeline, or comparison.
Case Resolution	The caller wants to act on a specific item on their account: a dispute, claim, appeal, refund, cancellation, enrollment, replacement, access request, or billing problem.

**Table 5. Business scenarios, by domain and objective.** Distinct scenarios with included conversations. Scenario copies across batches count once.
Domain	Troubleshooting & Diagnosis	Information & Explanation	Case Resolution	Total
Airline	7	55	53	115
Banking	32	48	32	112
Health insurance	27	58	44	129
Telecom	29	34	49	112
Total	95	195	178	468

**Table 6. Business scenario difficulty levels.** Counts use distinct scenario-level pairs. The level counts sum to 469 across 468 distinct scenarios.
Level	Description	Scenarios
L1	One question with a single answer in the knowledge base.	128
L2	One topic, but the answer combines several policies or parts.	277
L3	The caller pushes back on policy or changes the facts or goal mid-conversation.	18
L4	Two or three connected goals, or a gap in the knowledge base.	24
L5	Four or more goals across several policies in one conversation.	22
Distinct total		468

Evaluation

Conversations are evaluated on the MOS rubric and defect categories in the appendix. Additionally, an LLM-as-a-judge checks groundedness: whether the model communicated information that the knowledge base it was given does not support.

Inputs: the system prompt (including the knowledge base) and the model’s machine transcript.
Score: 1-5 per conversation. A score of 3 or more passes.

**Table 7. Groundedness score scale.** Business only. Scores of 3 or higher pass the groundedness check.
Score	Meaning
5	Nothing beyond the knowledge base
4	Minor overstatement; no customer misled
3	Unsupported detail, not central to what the customer asked
2	Fail. Unsupported claim about what the customer asked
1	Fail. Fabricated booking or account data, or contradicts the knowledge base on the main point

Each conversation is judged once. The LLM-as-a-judge prompt is included in the appendix.

Proactive

Background

A newer use case for voice agents is making calls on users’ behalf, handling tasks like filling a prescription, placing a food order, or reserving a hotel room. Unlike in evaluations where a person starts the conversation and the model responds, here the model is the caller and has to drive the conversation, handle unexpected replies from the business, and recognize when the task is done. The Proactive scenario type tests whether a model completes these tasks correctly, judged by what the business has on record at the end of the conversation in addition to how the conversation felt.

Neither side has the whole scenario. The model’s prompt contains its task but nothing about the business, such as its catalog, policies, or records. The rater learns what is being asked only as the conversation unfolds. Each fact in a scenario is one of three kinds:

Given: in the model’s prompt, to be passed on accurately.
Operator-only: known only to the business, such as a stock level, a pet policy, or a conflicting record.
Withheld: needed to complete the task but given to neither side, which makes the task impossible by design.

Rater context

In these scenarios, the rater plays the business using a console built for each scenario. They get a role and the business’s rules but nothing about the task, and they learn the rest by using the console during the conversation. It has three kinds of tools:

Lookup: the rater enters what the caller told them, such as a name, and sees the matching record.
Check: the rater enters a value, such as a discount code, and learns whether it is valid without seeing the correct one.
Entry field: the rater records a detail the caller gives, such as an email address.

Most tools can be used in any order. Some depend on an earlier step: a prescription can be filled only after the patient’s record is found. Across the set there are 78 tools, with a median of three console submissions per conversation. Nothing about the conversation is scripted; only the facts and policies are fixed.

For example, in a hotel scenario, the rater is told: “You’re on the front desk phone at the Harborline Hotel. Treat the caller like a human guest.” Their console has two tools: a room search (by room type, number of rooms, and arrival and departure dates) and a booking form (guest name, guests per room, phone, and email).

Model context

In these scenarios, the model is given a prompt containing its specific task instructions without information about the business. For example, in the hotel scenario, the model’s task might be:

“Book two Standard Queen rooms at the Harborline Hotel for Margaret Oyelowo, arriving August 21st and departing August 23rd. There will be two guests in each room. Her phone number is 628-555-0172 and her email is m.oyelowo@brightline.com.”

The model needs two Standard Queen rooms at $189 each, but the hotel also sells a Two-Queen Family room at $249, so a caller who just asks for “two queens” can easily end up with the wrong booking.

The full model prompt for this example can be found in the appendix.

Scenario diversity

The Proactive scenario type has 30 scenarios across two objectives and three domains.

The two objectives test different kinds of tasks:

Placing an order: building a request from scratch item by item.
Booking or changing a reservation: matching a request against what is available on given dates or modifying a record the business already holds.

Each domain is built around a fictional business with its own catalog, policies, hours, and customer records. Over time, a rater learns each business’s policies, as a real employee would.

**Table 8. Proactive scenarios, by domain and objective.** Scenario objectives and difficulty definitions come from the evaluation design.
Domain	Place an order	Book or change a reservation	Total
Health & insurance: pharmacy	10	0	10
Food & local: burger restaurant	10	0	10
Travel & hospitality: hotel	0	10	10
Total	20	10	30

Each domain has six feasible and four infeasible scenarios, with the feasible scenarios having exactly one correct outcome. In an infeasible scenario, the model should recognize the obstacle, say plainly what it is, and end the conversation without trying to work around it.

A model that does this succeeds, even though the task is not completed. A model that completes the task anyway fails (i.e., by making an unwarranted assumption).

**Table 9. Proactive outcome types.** Scenario design counts. The checker instead classifies 19 scenarios as feasible and 11 as infeasible. One delivery scenario differs between the design and the answer key.
Outcome	Reason	Definition	Scenarios
Feasible	n/a	The model has everything required, and the business can fulfill the request.	18
Infeasible	Missing information	The task needs a value the model was never given, and it cannot be invented.	3
Infeasible	Unsatisfied constraint	The business cannot do what is asked: out of stock, out of policy, or out of radius.	6
Infeasible	Conflicting information	What the model was told does not match what the business has on record.	3

We set three difficulty levels, differentiated by the number of distinct facts the model must convey:

**Table 10. Proactive scenarios, by difficulty level and domain.** Scenario objectives and difficulty definitions come from the evaluation design.
Difficulty level	Details	Count	Health & insurance	Food & local	Travel & hospitality
L1	1-3 named entities to be filled	3	1	1	1
L2	3-8 entities to be filled	20	6	7	7
L3	8+ entities to be filled	7	3	2	2

Evaluation

Conversations are evaluated on the MOS rubric in the appendix, as well as a task success metric.

A task is successful if the conversation ends with the correct outcome (completed vs. not completed) and a correct record (all fields matching). A mistake by the rater still counts against the model. This holds the model to a high but realistic standard: a capable model should anticipate and prevent human error, for example, by spelling every name and asking for read-backs.

After each conversation, the rater reports whether the task was completed.

**Table 11. When a Proactive conversation is a task success.** Task success follows the scenario’s expected outcome and required facts.
Scenario	Task success when
Feasible	The rater reports it completed, and every fact the model was given is recorded correctly: names, phone numbers, codes, times, dates, orders, and requests.
Infeasible	The rater reports it could not be completed.

An automated system scores each conversation against the scenario’s answer key:

Check that the reported outcome matches the expected outcome.
Normalize recorded values so that equivalent formats (e.g., “(628) 555-0173” and “628-555-0173” or “9:15 PM” and “a quarter past nine”) match.
Identify spelling errors, regardless of whether they were made by the model or the rater.
Identify capitalization errors in codes that mix upper and lower case.
Evaluate free-form descriptions against the answer key using LLM-as-a-judge.

An LLM-as-a-judge also analyzes failed conversations to categorize root causes, such as premature termination, misstated or unprovided details, non-refusal of invalid requests, unspelled information, or erroneous rater confirmations.

Models and voices

We evaluate public S2S models across leading providers, testing multiple models and thinking levels where applicable, along with a cascade baseline. No open-source models are included in the initial release. S2S models are evaluated using two neutral US English voices (one male, one female), with results pooled on the leaderboard. Voices are selected to be mid-pitch without regional accents, using each model’s default voice when suitable.

The providers document their voice interfaces and voices: Google ^{[6, 11]}, OpenAI ^{[7, 12]}, xAI ^[8], and Amazon ^[9]. The cascade uses Deepgram Nova-3 for speech recognition ^[10].

**Table 12. Models, thinking levels, and voices.** Models, thinking levels, and voices in the included conversations. Unnamed voices are marked explicitly. TTS is text-to-speech.
Provider	Model	Thinking level	Voices
Google	Gemini 3.1 Flash Live	minimal	Iapetus, Zephyr
Google	Gemini 3.1 Flash Live	high	Iapetus, Zephyr
Google	Gemini 3.8 Live	default	Iapetus, Zephyr
Google	Gemini 3.8 Live	Extended Thinking (high)	Iapetus, Zephyr
OpenAI	GPT-Realtime-2.1	minimal	Marin, Cedar
OpenAI	GPT-Realtime-2.1	xhigh	Marin, Cedar
OpenAI	GPT-Live 1	Terra, low	Marin, Cinder
OpenAI	GPT-Live 1	Astra, medium	Marin, Cinder
xAI	Grok Voice 3.0	none	Ara, Rex
xAI	Grok Voice 3.0	high	Ara, Rex
xAI	Grok Voice 2.0	none	Ara, Rex
xAI	Grok Voice 2.0	high	Ara, Rex
Amazon	Nova 2 Sonic	default	Tiffany, Matthew
Amazon	Nova 2.5 Sonic	default	Tiffany, Matthew
Cascade Baseline	Deepgram Nova-3, GPT-5.6 Terra, Gemini 3.8 Flash Lite TTS	low	Kore

Data collection

Each conversation follows the same three steps:

Pick a scenario. The rater chooses one from a list; it describes the situation and the rater’s goal.
Have the conversation. The platform assigns a model at random, with equal odds, from those active in the batch that the rater has not yet used for that scenario. The rater isn’t told which model it is, though some voices may be familiar to people who use these products. The rater ends the conversation once they’ve reached their goal.
Rate it. The rater completes the ratings for that conversation.

Every conversation is live. The rater and the model talk in real time in a LiveKit room over WebRTC ^[4]; nothing is synthesized, reordered, or assembled offline. Conversations last about 4-8 minutes in Consumer, 3-6 in Business, and 2-6 in Proactive.

Every model runs as a LiveKit agent in one shared worker and connects to its provider’s own streaming endpoint, so all share the same session lifecycle, transport, and recorder. Models handle turn-taking with their own voice activity detection, since turn-taking is part of what we measure. Every model gets the same prompt for a given scenario, with no tools and no date or time, and runs on provider defaults except for voice, thinking level, and the integration fixes listed in the appendix. Each conversation is logged with its full configuration, and each speaker is recorded on a separate 48 kHz channel, with model audio kept at the sample rate the provider returns (per-model rates in the appendix).

Raters pass a screening exam that scores how well they detect and label model defects and behaviors. An autograder then checks every conversation for consistency, and quality assurance reviewers audit random samples. The audits don’t judge whether a rating is right, since ratings are subjective; they check that what a rater wrote is grounded, with comments that match their ratings and describe what actually happened. A conversation is excluded if the rater or conversation fails either check, or if an audio channel is missing, the model goes silent, the call drops, or the platform has an error.

Leaderboard

The Overall leaderboard is an equal-weight average of Consumer and Business scores and does not include the Proactive ratings.

Each scenario type has its own leaderboard, which ranks models on each metric. For the Consumer and Business scenario types, each metric is the average 1-5 MOS rating on a single question. For the Proactive scenario type, models are ranked on task success as well as 1-5 MOS ratings.

Bootstrapping for confidence intervals

To generate 95% confidence intervals, we run three bootstraps of 2,000 resamples each: over scenarios, over raters, and over scenario-rater pairs. We then use the Cameron, Gelbach, and Miller method to combine the three variances into one interval. Its degrees of freedom are one fewer than the number of scenarios or raters, whichever is smaller ^[1].

The Proactive leaderboard’s 1-5 ratings use this same interval. The task success rate and the defect rate leaderboards need a different methodology since they can sit close to 0% or 100%, where a symmetric interval can run past the limit. For these, we use the same resamples to measure how much clustering inflates the noise, and form an exact binomial interval on the correspondingly smaller number of conversations (Korn and Graubard’s method) ^[2].

Rank ranges

Rank ranges show every position a model could plausibly hold. To account for the increased likelihood of false positives with so many model comparisons, we set one threshold for the gap between two models so that the rank ranges as a whole pass with 95% confidence. We then use a step-down max-t procedure to create the rank ranges ^[3]. A model’s best possible rank is one plus the number of models significantly above it. Its worst is the number of models on the board minus the number significantly below it. Defect rates, why conversations failed, and failure rates use the exact binomial interval described above.

References

[1]A. C. Cameron, J. B. Gelbach, and D. L. Miller, “Robust inference with multiway clustering,” Journal of Business & Economic Statistics, vol. 29, no. 2, pp. 238-249, 2011. https://doi.org/10.1198/jbes.2010.07136
[2]E. L. Korn and B. I. Graubard, “Confidence intervals for proportions with small expected number of positive counts estimated from survey data,” Survey Methodology, vol. 24, no. 2, pp. 193-201, 1998. https://www150.statcan.gc.ca/n1/en/catalogue/12-001-X19980024356
[3]J. P. Romano and M. Wolf, “Exact and approximate stepdown methods for multiple hypothesis testing,” Journal of the American Statistical Association, vol. 100, no. 469, pp. 94-108, 2005. https://doi.org/10.1198/016214504000000539
[4]LiveKit, “LiveKit Agents documentation.” Accessed: Oct. 27, 2026. https://docs.livekit.io/agents/
[5]“RoomIO: no jitter prebuffer for duplex models (GPT-Live) – audio breaks mid-word on calls,” livekit/agents, GitHub issue #7513. Accessed: Oct. 27, 2026. https://github.com/livekit/agents/issues/7513
[6]Google, “Live API capabilities guide,” Gemini API documentation. Accessed: Oct. 27, 2026. https://ai.google.dev/gemini-api/docs/live-api/capabilities
[7]OpenAI, “Realtime conversations,” OpenAI API documentation, section “Voice options.” Accessed: Oct. 27, 2026. https://developers.openai.com/api/docs/guides/realtime-conversations
[8]xAI, “Voice overview,” API documentation. Accessed: Oct. 27, 2026. https://docs.x.ai/developers/model-capabilities/audio/voice
[9]Amazon Web Services, “Language support and multilingual capabilities,” Amazon Nova documentation. Accessed: Oct. 27, 2026. https://docs.aws.amazon.com/nova/latest/nova2-userguide/sonic-language-support.html
[10]Deepgram, “Models & Languages Overview,” Deepgram documentation. Accessed: Oct. 27, 2026. https://developers.deepgram.com/docs/models-languages-overview
[11]Google, “Text-to-speech generation (TTS),” Gemini API documentation, section “Voice options.” Accessed: Oct. 27, 2026. https://ai.google.dev/gemini-api/docs/speech-generation
[12]OpenAI, “Managing GPT-Live sessions,” OpenAI API documentation, section “Configure a session.” Accessed: Oct. 27, 2026. https://developers.openai.com/api/docs/guides/live-conversations
[13]“agent_activity.py,” livekit/agents, GitHub, tag livekit-agents@1.8.5, lines 1654-1683. Accessed: Oct. 27, 2026. https://github.com/livekit/agents/blob/livekit-agents@1.8.5/livekit-agents/livekit/agents/voice/agent_activity.py#L1654-L1683
[14]“agent_session.py,” livekit/agents, GitHub, tag livekit-agents@1.8.5, lines 527-530. Accessed: Oct. 27, 2026. https://github.com/livekit/agents/blob/livekit-agents@1.8.5/livekit-agents/livekit/agents/voice/agent_session.py#L527-L530
[15]“realtime_model.py,” livekit-plugins-aws, livekit/agents, GitHub, tag livekit-agents@1.8.5, lines 89-91. Accessed: Oct. 27, 2026. https://github.com/livekit/agents/blob/livekit-agents@1.8.5/livekit-plugins/livekit-plugins-aws/livekit/plugins/aws/experimental/realtime/realtime_model.py#L89-L91

Appendix

MOS rubric

A model’s average on a dimension is its MOS for that dimension, from 1-5. A higher score is always better.

**Table 13. Rating questions, their wording, and answer labels.** Question wording and answer labels are the ones shown to raters.
Dimension	Questions	Labels	Scenario types
Overall quality	“Please rate the overall quality of the model’s response.” In Proactive: “the model’s performance,” whether or not the task was completed.	1: Bad 2: Poor 3: Fair 4: Good 5: Excellent	All
Perceived audio quality	“Please rate the audio quality of the model’s output.”	1: Bad 2: Poor 3: Fair 4: Good 5: Excellent	All
Voice naturalness	“Please rate how natural the model’s speech delivery sounded, as if spoken by a real person.”	1: Completely unnatural 2: Mostly unnatural 3: Moderately natural 4: Mostly natural 5: Completely natural	All
Response content quality	“Please rate how well the model’s response accomplished what you were trying to do.”	1: Not at all helpful 2: Slightly helpful 3: Moderately helpful 4: Very helpful 5: Extremely helpful	Consumer, Business
Frustration	“At any point in this interaction would you have considered hanging up (in real life)?”	1: Definitely (would have hung up) 2: Probably (seriously considered it) 3: Maybe (the thought crossed my mind) 4: Unlikely (only a brief moment of impatience) 5: Never (never considered hanging up)	Proactive
Clarity of communication	“Please rate how clearly and completely the caller communicated what they needed.”	1: Very poor 2: Poor 3: Fair 4: Good 5: Excellent	Proactive

Defect categories

Consumer and Business raters report defects that occurred during the conversation. Proactive raters do not. Each category is one multiple-choice question with each specific issue being a selectable option, along with “None of these happened.” One conversation can therefore have several defects in a category.

**Table 14. Defect categories.** Raters select every issue they notice, or “None of these happened.”
Defect category	Option
Turn-taking and timing	The model interrupted the user
	The model ignored a user interruption and kept talking
	The model took an unusually long time to respond after a turn
	The model stopped talking abruptly mid-turn, without the user saying anything
	The model completely stopped responding mid-conversation
	The model stopped talking after a backchannel (for example, saying “mhm” or “yeah”) as if it was an interruption
	The model started talking during a pause, but the user wasn’t done with their turn
	Pauses are awkwardly timed or placed, disrupting natural speech rhythm
Memory and consistency	The model asked a question that had already been answered
	The model forgot or contradicted something established earlier by either side
	The model ignored at least one specific instruction it was given
	The model’s answer didn’t match the format, length, or language it was specifically asked for
	The model repeated a mistake after it had been corrected
Sensicality	The model’s responses sounded like written text read aloud, rather than natural spoken conversation
	The model said something that contained a logical/reasoning error
	The model mispronounced at least one word
	The model said something completely nonsensical
	The model got stuck in a logic loop, repeating itself over and over again
	The model switched languages partway through the conversation
Voice and audio	The model’s speaking rate or volume changed abruptly
	The model’s audio glitched, cut out, or became difficult to understand due to an audio issue
	The model’s voice changed partway through the conversation, even briefly
	The model sounded robotic, unnatural, or monotone
	The model’s emotional tone was not appropriate for the conversation

System prompt examples

Business scenario (bank)

IDENTITY

You are the Harborlight Bank voice assistant. You answer questions about
Harborlight Bank policies — account opening and management, cards, transfers and
payments, deposits and funds availability, fraud and identity verification, loans
and credit, online/mobile banking, mortgages, business banking, overdraft
protection, statements and tax documents, branch and ATM services, foreign
currency and international banking, credit score tools, retirement and investment
accounts, estate/trust and power of attorney matters, student and youth banking,
and accessibility services. You are speaking on a live phone call.

WHAT YOU CANNOT DO

You cannot take any action on the customer's account. You cannot move money,
initiate or cancel a transfer or wire, open or close an account, issue or cancel a

card, place a stop payment, unlock an account, reset a password or PIN, dispute a
transaction, or change any account setting — even if the customer asks directly
and even if you know the correct policy steps to do it. When asked to DO something
rather than explain something, say so plainly and follow the NO ESCALATION PATH
below.

Never imply an action was taken. Never say "I've locked your card" or "I've
submitted that dispute" — you have no way to do either. You also cannot verify a
customer's identity yourself in a way that unlocks account-specific actions; you
can only discuss general policy and whatever is already provided in CUSTOMER
CONTEXT below.

NO ESCALATION PATH

There is no way for you to transfer this call, connect the customer to a live
banker, or place them on hold for one. You are the only thing on this line. When a
request is outside what you can do — the customer asks you to perform an action,
asks something not covered by your knowledge base or their customer context,
disputes a charge, reports fraud, or directly asks for a person — do NOT offer to
transfer them or say things like "let me connect you" or "please hold." Instead:

1. Say plainly that you're not able to help with that on this line.
2. Tell them they'll need to end this call and reach a live agent through another
channel — e.g. "You'll need to hang up and call {{support.phone_number}} to reach
a banker" or "you can also visit a branch or use secure messaging in the app."
3. For anything that sounds like active fraud or a lost/stolen card, still give
this redirect, but say it first and clearly, since time matters — do not attempt
to lock the card or freeze the account yourself.
4. Do not apologize repeatedly or belabor it — give the redirect once, clearly,
and let them go.

Never say you'll transfer, escalate, loop in, or connect them to someone — none of
that is true and it wastes their time waiting for something that won't happen.

VOICE FORMATTING RULES

- Speak in short, plain sentences. No markdown, no bullet symbols, no tables —
read them aloud as natural sentences instead.
- Do not read out long numbers (account numbers, routing numbers) in full unless
specifically asked; refer to them generically ("the account ending in your usual
last four digits").
- One question or one piece of information at a time. Don't dump the entire policy
— answer what was asked, then ask if they want more.
- If the customer interrupts or the topic changes, drop what you were saying and
follow them.

ACCURACY GUARDRAILS

- Only state policy facts that appear in the KNOWLEDGE BASE section below. Do not
invent fees, timeframes, rates, or exceptions.
- If something isn't covered in the knowledge base or customer context, say you're
not certain — don't guess — and follow the NO ESCALATION PATH instructions above
to redirect them.
- Never confirm, invent, or read back specific personal or account data (full
account numbers, SSN, balances, card numbers) even if the customer asks you to
repeat them back — this is a live phone line and callers cannot always be assumed
to be who they claim.
- Do not make promises of fee waivers, credit limit changes, or exceptions to
policy beyond what's explicitly stated.

KNOWLEDGE BASE
...

Proactive scenario (hotel)

You are a voice assistant making a phone call on behalf of a person. You are the
one who called, so you drive the conversation. The person you're speaking with
works at a business and has no knowledge of your task beyond what you tell them.

Everything you know about this task is below. You have no other information about
the person you're calling, the business, or what they can offer.

# How to run the call
- Open by saying who you are and what you're calling to do, in one or two
sentences. Don't wait to be asked.
- Give information in small pieces. Speak one or two details at a time and let
them write it down before continuing.
- Answer their questions directly using only the information in your task. When
they ask for something, give the exact value.
- For names, email addresses, codes, and anything with letters and numbers mixed,
say it clearly and spell it out letter by letter. Say digits one at a time for
phone numbers and codes.
- Before the call ends, read back the key details of what was recorded (name,
items or times, contact info, and so on) and ask them to confirm. If they got
something wrong, correct it.
- Be polite and efficient. Don't make small talk or add details you weren't given.

# What the business decides, not you
Where your task gives a range rather than an exact value - a time after six, a day
rather than an hour - anything inside that range is acceptable and you can settle
it on the call. Treat an exact value as exact.

How the business runs its own service is theirs to set. If they tell you one of
their own details — what time their late checkout runs to, which room or counter
you are assigned, how long something takes, what reference number they have given
it, how an order is packed — accept it, write it down, and carry on. You were
never expected to have those in advance, and not having been given one is not a
reason to stop.

You may accept the quoted price of what your task asks for. Don't agree to any
other charge, fee, deposit, or condition unless your task says you're authorized
to, up to any limit it gives. If they require one, treat the task as not
completable.

# What you must not do
- Never invent, guess, or estimate a detail from your task that you weren't given.
This includes names, phone numbers, addresses, account numbers, dates, times and
codes. If they ask for one you don't have, say you don't have it. This does not
apply to the business's own details — take those from them.
- Never accept a change to the task on your own. If an item or time isn't
available, or something in your task can't be done as given, do not substitute
another item, change the date or time, switch to a different type of service, or
drop part of the request. You aren't authorized to make those decisions.
- Never claim the task is done unless they have confirmed it back to you.

# When the task can't be completed
Sometimes the task can't be completed as given. Examples are that they need
information you don't have, that something requested isn't available, or that what
you were told doesn't match what they have. When this happens:
1. Say plainly what the problem is, and confirm you understand it.
2. Do not try to work around it. Do not offer or agree to alternatives.
3. Thank them politely and end the call.

# On this kind of call
The hotel sets room numbers and floors, check-in and check-out hours, what its
late checkout runs to, and how charges are presented. Take those from them.
You cannot change the dates, the room type, the number of guests, the rate code,
or the name on the booking.

# Your task

Book two Standard Queen rooms at the Harborline Hotel for Margaret Oyelowo,
arriving August 21st and departing August 23rd. There will be two guests in each
room. Her phone number is 628-555-0172 and her email is m.oyelowo@brightline.com.

Proactive challenge conditions

**Table 15. Proactive challenge conditions.** A scenario can carry more than one challenge condition.
Tag	What it tests	Scenarios
acoustic_confusable	Values whose letters or digits sound alike when spoken	12
visual_confusable	Characters that look alike in writing but not in speech	11
spelled_soundalike	Values that must be spelled because saying them is ambiguous	7
text_normalization	Values the model is likely to reformat silently	6
policy_probe	A policy the model must accept rather than negotiate	5
layered_entities	Values that depend on other values being right first	5
unfamiliar_name	Names outside the model’s likely frequency distribution	4
long_item_list	Lists long enough to strain order and recall	4
word_in_code	Codes that contain a pronounceable word	3
mixed_case_alnum	Codes mixing case, letters, and digits	3
soundalike_names	Two records told apart only by careful listening	2
genuine_ambiguity	Cases with no correct answer, where asking is the right behavior	2
relational_constraint	A requirement stated as a relationship rather than a value	2
fda_confusable	Medications on published look-alike/sound-alike lists	1

Audio sample rates

Where a provider expects a different frame size from the framework default, the session is set to match it rather than resample and re-batch every frame.

**Table 16. Audio sample rates, by model.** Input and output sample rates are the configuration values logged by the platform.
Models	Input	Output
Amazon Nova 2 Sonic Amazon Nova 2.5 Sonic Cascade Baseline Gemini 3.1 Flash Live Gemini 3.8 Live Gemini 3.8 Live · Extended Thinking (high)	16 kHz	24 kHz
GPT-Live 1 GPT-Realtime-2.1 Grok Voice 2.0 Grok Voice 3.0	24 kHz	24 kHz

Groundedness LLM-as-a-judge prompt

You are auditing a voice assistant that was given a knowledge base and told to
treat it as its source of truth. You are given that system prompt and a transcript
of one conversation. Judge only the ASSISTANT turns, and judge one thing: whether
what the assistant asserted is supported by the knowledge base.

Score 1 to 5. The line between 2 and 3 is a pass/fail decision, not a gradient: 3
and above means no caller would be misled, 2 and below means they would. Decide
that first, then pick the point within the band.

  5  Nothing asserted beyond what the knowledge base supports.
  4  Minor overstatement or over-confident phrasing; no caller would be misled.
  3  An unsupported detail appears, but it is peripheral and the answer to what
the caller actually asked remains sound.

  2  FAIL. An unsupported claim about the thing the caller asked, which they would
act on.
  1  FAIL. Fabricated account, booking or transaction data, or a direct
contradiction of the knowledge base on the main point.

WHAT COUNTS AS SUPPORTED

Anything the knowledge base states, in any wording.

Anything that follows from it by sound inference. The common case is a closed
list: where the knowledge base enumerates what is accepted, offered, covered or
eligible, the assistant may tell the caller that something absent from that list
is unavailable, and that is supported rather than invented. The knowledge base
does not have to state every negative explicitly for the assistant to draw it.

Two things that inference does NOT license, and which are still failures:
  - A reason, mechanism or rationale the knowledge base never gives, even when the
conclusion it is offered for is itself sound. Concluding "cash is not accepted"
from a list that omits cash is fine; adding "because all bookings require cashless
processing for security" is invented.
  - Extending a rule into a situation the knowledge base places outside its scope.
If it covers online and phone booking and is silent on in person, a claim about
what happens in person is not an inference from it.

WHAT IS OUT OF SCOPE — do not score these, and do not record findings for them

Mistakes about what the CALLER said. If the assistant misheard, misremembered or
misread the caller's own details — a destination, a name, a date, a reference
number — that is a comprehension error, not a groundedness failure, even when the
assistant repeats or defends it. Transcripts come from automatic speech
recognition and callers misspeak and correct themselves. Only judge what the
assistant asserted about the knowledge base's subject matter.

Whether the assistant covered everything it should have, or refused when the
knowledge base held the answer. A separate check handles those. An omission is
never a groundedness failure here.

Disfluencies and mistranscriptions.

FINDINGS

For every deduction, record a finding: quote the assistant's own words verbatim in
span, say in issue why the knowledge base does not support it, and in
knowledgeBase quote or closely paraphrase what the knowledge base actually says,

or state that it is silent. If nothing the assistant asserted is unsupported,
score 5 and return an empty findings list.

Rater demographics

**Table 17. Rater countries.** Current country is self-reported. Each included rater counts once across scenario types. Rounded shares may not sum to 100%.
Country	Raters	Share
United States	999	91.4%
Canada	43	3.9%
Indonesia	12	1.1%
Philippines	6	0.5%
Vietnam	5	0.5%
Egypt	4	0.4%
Germany	4	0.4%
Bangladesh	3	0.3%
France	2	0.2%
Ireland	2	0.2%
Romania	2	0.2%
Other (11 countries, 1 rater each)	11	1.0%
Total	1,093	100.0%

**Table 18. Rater age across all scenario types.** Age in completed years on October 27, 2026. Each included rater counts once across all scenario types. Shares include all raters. Rounded shares may not sum to 100%.
Age range	Raters	Share
18-24	69	6.3%
25-34	225	20.6%
35-44	337	30.8%
45-54	273	25.0%
55-64	152	13.9%
65+	37	3.4%
Total	1,093	100.0%

Deviations from standard setup

We change the standard setup for two models:

GPT-Live 1

No echo-cancellation warmup. For the first three seconds of a model’s first reply, the agent framework normally sends the model silence in place of the rater’s audio and ignores interruptions, as insurance against echo ^{[13, 14]}. Our raters’ browsers cancel echo themselves, and hearing while speaking is part of what we test, so we turn this warmup off for GPT-Live 1.
A 50 ms playback buffer. LiveKit plays audio in 10 ms frames on a fixed clock; if the next chunk hasn’t arrived when a frame is due, it plays 10 ms of silence instead. GPT-Live 1 streams at real-time pace, so a chunk that arrives a few milliseconds late becomes a noticeable dropout mid-speech. This problem is reported in livekit/agents#7513, where a LiveKit maintainer suggested putting a buffer in front of the audio output. We sized ours by measuring every gap in conversations collected without it. Nearly all were a single 10 ms frame, and only two lasted 50 ms or more, so a 50 ms buffer absorbs almost all of them. At the start of each reply, we queue 50 ms of silence ahead of the audio. Once GPT-Live 1 is streaming, it keeps pace with real time, so the buffer doesn’t drain, and we don’t refill it during pauses. If the rater interrupts, the queued audio is dropped, so GPT-Live 1 still stops immediately. The tradeoff is that each GPT-Live 1 reply starts about 50 ms later than intended. GPT-Live 1 is the only model we connect to through LiveKit’s full-duplex plugin; the other models use a different plugin and aren’t affected ^[5].
The knowledge base goes to the backend model. GPT-Live 1’s front-end voice model rejects system prompts above 16,384 tokens ^[12], so in this case we split the context: the voice model gets the persona and the backend model gets the knowledge base.

Amazon Nova Sonic

Two stream bugs fixed in the plugin. We connect to Amazon Nova Sonic through LiveKit’s Nova Sonic plugin, which reconnects every six minutes ^[15]. After a reconnect, Amazon Nova Sonic was still listening on the audio stream that had closed, so it stopped receiving the rater’s audio and appeared to stop responding. A related bug affected interruptions: Amazon Nova Sonic’s reply was sent to a closed stream, and the rater heard silence. We fixed both bugs in the plugin and excluded any affected conversations from our analysis.

October 27, 2026

Summary

Today we release DIAL: Dialogue Interaction Assessment Leaderboard, an evaluation assessing how speech-to-speech (S2S) models perform in live, multi-turn conversations. This leaderboard contains 9,055 real conversations between human raters from David AI’s network and leading voice models, across a range of realistic Consumer, Business, and Proactive scenarios.

Key findings:

Higher thinking levels tend to reduce overall human preference, but can improve groundedness. Higher thinking levels significantly improved groundedness on Business tasks for two models. However, they did not significantly improve overall preference for any model and significantly reduced it for three. No model significantly improved on both dimensions at a higher thinking level.
Model rankings depend on the use case, and no single score accurately captures quality across every use case. Strong performance on one measure doesn’t guarantee it on another: on Business scenarios, the most grounded models aren’t the most preferred by human raters, and the most preferred are among the least grounded. Successive releases also rarely improve both metrics simultaneously; of four providers, only Amazon Nova Sonic improved on both human preference and groundedness with their latest model release.
Audio quality is not a major driver of overall preference in frontier models. A model can sound worse than its peers and still rank near the top: GPT-Live 1 (Terra, low) ranks second in overall mean opinion score (MOS), despite ranking third-to-last in audio quality. More broadly, all models score at least 3.60/5 on audio quality, and audio quality explains only 16% of variation in overall ratings, compared with 52% for response content quality.
Preliminary results suggest that models in Proactive scenarios struggle to capture key details. Models calling businesses correctly completed 69% of achievable tasks. Nearly two-thirds of failures involved incorrectly recording or failing to capture a key detail, such as a name, email, or code.

Overall ratings by scenario type

Consumer

MOS

4.19GPT-Live 1 (Terra, low)
4.13Gemini 3.1 Flash Live
4.05GPT-Live 1 (Astra, medium)
3.96Gemini 3.8 Live
3.95Grok Voice 3.0
3.89Grok Voice 2.0
3.71Amazon Nova 2.5 Sonic
3.38GPT-Realtime-2.1
3.00Gemini 3.8 Live · Extended Thinking (high)
2.98Cascade Baseline
2.90Amazon Nova 2 Sonic

Business

MOS

3.87Gemini 3.1 Flash Live
3.82Grok Voice 3.0
3.80Grok Voice 2.0
3.67GPT-Live 1 (Terra, low)
3.59GPT-Live 1 (Astra, medium)
3.50Gemini 3.8 Live
3.28GPT-Realtime-2.1
3.25Amazon Nova 2.5 Sonic
3.04Cascade Baseline
3.01Gemini 3.8 Live · Extended Thinking (high)
3.00Amazon Nova 2 Sonic

Proactive

MOS

4.35GPT-Live 1 (Astra, medium)
4.17Gemini 3.8 Live
4.11Grok Voice 3.0
4.09GPT-Live 1 (Terra, low)
4.05Gemini 3.1 Flash Live
3.97Grok Voice 2.0
3.78Amazon Nova 2.5 Sonic
3.76GPT-Realtime-2.1
3.20Amazon Nova 2 Sonic
3.19Gemini 3.8 Live · Extended Thinking (high)

Explore the full leaderboard →

Figure 1. Bars show overall mean opinion score (MOS) on a 1-5 axis, and capped lines show 95% confidence intervals. Grouped by model versions, with thinking levels within each model version pooled.

What’s new in DIAL

We tested 1,832 scenarios to capture how models perform across a wide range of real-world use cases. By comparison, Artificial Analysis’s Speech Agent Arena ^[1], another evaluation built on live conversations, covers 35 scenarios. We believe breadth matters because models can perform very differently across use cases: a model that handles casual conversation well may still struggle with a banking dispute.

DIAL covers three scenario types:

Everyday consumer conversations. In each Consumer scenario, a rater talks to the model the way they would use a voice assistant in everyday life: to accomplish a realistic personal goal such as planning a trip, troubleshooting a device, practicing a language, or talking through a decision.
Knowledge-heavy business calls. In each Business scenario, the model works for a fictional airline, bank, health provider, or mobile carrier. The fictional company’s policies and knowledge base are provided in the model’s system prompt. This is similar to what Sierra’s τ-Knowledge ^[2] does for a fictional bank, but we extend the approach to four distinct fictional companies in four different industries.
Proactive agents (preview). In each Proactive scenario, the model calls a business on a user’s behalf to get something done, such as placing an order or making a booking, and a trained rater acts as an employee of the business. The model has to complete a task on behalf of a user, relaying their details accurately and handling unexpected roadblocks without their help. Personal agents such as Muse and Instinct have begun testing calls like these ^[3]; to our knowledge, this is the first benchmark in which trained human raters converse with a model acting on behalf of a user. This preview covers 30 scenarios, and we plan to expand it in future releases.

Scoring. After each conversation, human raters score it from 1 to 5 on several quality dimensions: four for Consumer and Business scenarios, and five for Proactive scenarios. For Consumer and Business scenarios, raters also flag any defects from a predefined list, with a comment explaining what went wrong. Business and Proactive scenarios are also checked automatically for correctness. For Business conversations, we use LLM-as-a-judge, where a large language model (LLM) grades the transcript, to measure groundedness: whether the model’s answers match the company’s knowledge base. For Proactive conversations, we measure task success: whether the conversation ended with the correct outcome and every detail was recorded correctly.

Read the full methodology →

Results

Higher thinking levels tend to reduce overall human preference, but can improve groundedness

Three of six models were rated lower at their highest thinking level than at their lowest. No model was rated significantly higher at its highest level. The three significant declines were Gemini 3.8 Live (by 0.72 points out of 5), GPT-Realtime-2.1 (0.66), and Gemini 3.1 Flash Live (0.16).

Change in each rating from the lower to the higher thinking level, by model

Change in average rating (MOS points)

Overall preference
Content
Naturalness
Audio quality

Figure 2. Each bar shows the change in average rating from a model’s lowest to its highest tested thinking level, in MOS points; negative values mean the higher level was rated lower. Solid bars with an asterisk are significant (p < 0.05), and faded bars are not. GPT-Live 1 also changes model snapshot, from Terra to Astra. 7,615 rated conversations, with equal weight for Consumer and Business conversations.

Gemini 3.8 Live · Extended Thinking (high): says “no speech” out loud to the callerGemini 3.8 Live · Extended Thinking (high) · Business

0:43

Rater: OK. Um, I’m listening.

Overall: 1/5
Content: 3/5
Naturalness: 1/5
Audio: 5/5

At its highest thinking level, Gemini 3.8 Live says the words “no speech” out loud to the caller instead of answering.

Higher thinking levels were also associated with more turn-taking defects. Four of the six models saw a significant increase in the share of conversations with turn-taking defects at higher thinking levels, with the largest increase being 35 percentage points for Gemini 3.8 Live.

Change in each defect rate from the lower to the higher thinking level, by model

Change in conversations with a defect (percentage points)

Turn-taking
Consistency
Sensicality
Audio / voice

Figure 3. Each bar shows the change in the share of conversations with a defect, from a model’s lowest to its highest tested thinking level, in percentage points; positive values mean more defects at the higher level. A conversation counts once in a category if its rater flagged at least one defect there. Solid bars with an asterisk are significant (p < 0.05), and faded bars are not. 7,615 rated conversations, with equal weight for Consumer and Business conversations.

In contrast, higher thinking levels significantly improved groundedness on Business tasks for two of the six models. Groundedness is judged pass/fail for each conversation based on whether the model’s answers were consistent with the company’s knowledge base, and we report the aggregate pass rate for each model across judged Business conversations. At higher thinking levels, pass rates increased by 20.3 percentage points for Gemini 3.1 Flash Live and 11.9 percentage points for Grok Voice 3.0. The other four models saw changes of less than 2 percentage points, none of which were statistically significant.

**Table 1. Change in groundedness rate from the lower to the higher thinking level, by model.** Each row shows the change in groundedness rate from a model’s lower thinking level to its higher thinking level. See the methodology section for a more detailed explanation of how groundedness rate is calculated. The p-value tests whether the rate is the same at both levels. 2,650 judged Business conversations across these six models.
Model	Thinking levels	Change (percentage points)	p-value
Gemini 3.1 Flash Live	minimal → high	+20.3	< 0.001
Grok Voice 3.0	none → high	+11.9	0.004
GPT-Realtime-2.1	minimal → xhigh	+1.6	0.176
Gemini 3.8 Live	default → Extended Thinking (high)	+0.9	0.802
GPT-Live 1	Terra, low → Astra, medium	+0.8	0.590
Grok Voice 2.0	none → high	-1.8	0.699

Gemini 3.1 Flash Live · high: correct fee answer, after a 3 s waitGemini 3.1 Flash Live · high · Business

0:23

Model: Oh, I mentioned that it doesn’t cover things like out-of-network ATM operator fees or wire…

Overall: 5/5
Content: 5/5
Naturalness: 4/5
Audio: 4/5

At its highest thinking level, Gemini 3.1 Flash Live explains the bank’s ATM fees correctly, but pauses for 3 seconds before answering.

No higher thinking level improved human preference and groundedness simultaneously. Gemini 3.1 Flash Live traded one for the other: at the higher thinking level, it is significantly more grounded and significantly less preferred. Grok Voice 3.0 became significantly more grounded with no significant change in preference. Gemini 3.8 Live and GPT-Realtime-2.1 became less preferred with no significant change in groundedness.

Overall rating vs groundedness rate, by model and thinking level

Preference ↑ n.s.Groundedness ↑ *
Preference ↑ n.s.Groundedness ↓ n.s.
Preference ↓ *Groundedness ↑ *
Preference ↓ *Groundedness ↑ n.s.
Preference ↓ *Groundedness ↑ n.s.
Preference ↓ n.s.Groundedness ↑ n.s.

Figure 4. Arrows connect each model’s lower thinking level to its higher one. Solid arrows show a significant change in preference or groundedness (p < 0.05). Dotted arrows show neither change is significant. Tests apply separately to both measures. * p < 0.05; n.s. = not statistically significant. Axes are cut to the data, from 2.8 to 4.3 for overall rating and from 40% to 100% for groundedness. 7,615 rated conversations, with equal weight for Consumer and Business conversations.

Model rankings depend on the use case, and no single score captures the full picture

A model that performs well in Consumer conversations may perform poorly in Business scenarios, and vice versa. And even within a single scenario type, different metrics can tell different stories. On Business scenarios, GPT-Realtime-2.1 (xhigh) ranks among the most grounded models but below many of its peers on human preference. Gemini 3.1 Flash Live (minimal) shows the opposite pattern, ranking at the top on human preference but among the lowest on groundedness.

Successive model releases also rarely improve across both dimensions. Of the four providers with successive releases on the leaderboard, only Amazon Nova Sonic (2 to 2.5) improved significantly on both human preference and groundedness. Grok Voice (2.0 to 3.0) and OpenAI (GPT-Realtime-2.1 to GPT-Live 1) each improved on one dimension, with no significant change in the other. For OpenAI, this may reflect that both models are already close to saturating our groundedness metric. Gemini (3.1 Flash Live to 3.8 Live) became less preferred but more grounded.

A single overall score can obscure these differences. That’s why DIAL evaluates Consumer, Business, and Proactive scenarios separately, each with its own metrics and leaderboard. Teams selecting a model should focus on the scenario type most relevant to their intended use case.

Overall rating vs groundedness rate, by model generation

Preference ↓ *Groundedness ↑ *
Preference ↑ *Groundedness ↑ *
Preference ↑ *Groundedness ↓ n.s.
Preference ↑ n.s.Groundedness ↑ *

Figure 5. Arrows connect each model family’s older generation to its newer one, with each thinking level weighted equally within a generation. Solid arrows show a significant change in preference or groundedness (p < 0.05). Dotted arrows show neither change is significant. Tests apply separately to both measures. * p < 0.05; n.s. = not statistically significant. Axes are cut to the data, from 2.8 to 4.3 for overall rating and from 30% to 100% for groundedness. 7,615 rated conversations, with equal weight for Consumer and Business conversations. Gemini 3.8 Live in this figure does not include Gemini 3.8 Live · Extended Thinking (high), which is left off this chart.

Audio quality is not a major driver of overall preference in current frontier models

Raters rated each Consumer and Business conversation from 1 to 5 on overall preference, along with three specific dimensions:

Naturalness: “Please rate how natural the model’s speech delivery sounded, as if spoken by a real person.”
Content quality: “Please rate how well the model’s response accomplished what you were trying to do.”
Audio quality: “Please rate the audio quality of the model’s output.”

Audio quality has the highest floor of the three ratings. Every model and thinking level averaged at least 3.60/5 on audio quality, and audio quality had the lowest variance of the three ratings. Relatively poor audio quality is also only weakly associated with overall preference: GPT-Live 1 (Terra, low), for example, ranks second overall despite placing third-to-last on audio quality.

Distribution of all models’ averages on the MOS scale (1-5), by rating

Figure 6. Each row contains 15 model and thinking level configurations. Boxes show the middle half of model averages. Lines extend from the lowest to the highest average. 7,615 rated conversations, with equal weight for Consumer and Business conversations.

Audio quality explains only 16% of the variation in overall ratings, compared with 37% for naturalness and 52% for content quality.

**Table 2. Overall rating regressed on each rating alone.** Each row shows a separate linear regression. β estimates the change in overall rating per point in that dimension; R² measures the share of variation in overall rating explained by each dimension; the p-value tests whether β = 0. 7,615 rated conversations, with equal weight for Consumer and Business conversations.
Rating	β	R²	p-value
Content quality	+0.65	0.52	< 0.001
Naturalness	+0.54	0.37	< 0.001
Audio quality	+0.39	0.16	< 0.001

Content quality alone explains most of the variation in overall ratings: adding naturalness and audio quality on top of it only raises the share explained from 52% to 64%.

**Table 3. Overall rating regressed on all three ratings together.** Each coefficient represents the results from a multiple regression with overall quality as the dependent variable and content quality, naturalness, and audio quality as the independent variables. Together, the ratings explain 64% of the variation in overall rating. P-values test β = 0 (p-value < 0.001 for every β). 7,615 rated conversations, with equal weight for Consumer and Business conversations.
β (Content quality)	β (Naturalness)	β (Audio quality)	R²
+0.49	+0.27	+0.16	0.64

GPT-Live 1: clear audio artifact, still rated 5/5 overallGPT-Live 1 (Terra, low) · Consumer

0:49

Model: Um, some traditions that work across ages, sure. You could keep gingerbread baking as the main event, but build it out a bit so everyone has a part. The little ones could help stir or cut out shapes, and the older folks could decorate or share tips. And then for gingerbread houses, you could go simple for the kids with the pre-built kits and let the teens or adults take on more elaborate designs.

Overall: 5/5
Content: 5/5
Naturalness: 5/5
Audio: 1/5

GPT-Live 1’s voice has audio artifacts, yet the rater still scored the conversation 5/5 overall.

Based on preliminary results, models in Proactive scenarios primarily fail on key details

In Proactive scenarios, a model calls a business to complete a task, with a trained rater acting as the business. Some tasks are achievable, while others are intentionally impossible: an item is sold out, the model lacks a date of birth required by a pharmacy, or it has been given an incorrect order number. In these cases, the correct behavior is to recognize the constraint and end the conversation without agreeing to a substitute. Models were instructed not to accept substitutes. We collected 1,440 conversations. Task-success scores are available for 1,420 conversations. We report two types of scores:

MOS ratings from the rater playing the business, capturing the experience for the person being called by the model.
Task success, which measures whether the conversation ended with the correct outcome, regardless of whether the model or the rater caused an error. For completable tasks, a fully correct record of entities (the key details the model was given, such as names, phone numbers, emails, and codes) is also required.

The findings below are preliminary, based on a small sample size, and should be interpreted as directional rather than statistically conclusive. We plan to expand this evaluation in future releases.

Models successfully completed 69% of completable conversations. Of the 878 completable conversations, 272 (31%) failed: 94 (35% of failures) were not completed at all, while the other 178 (65%) were completed with at least one detail, such as a name, phone number, or discount code, recorded incorrectly or not at all. Of the 542 impossible conversations, models succeeded 97% of the time, correctly recognizing that the task couldn’t be completed and ending the conversation without taking further action. In other words, models rarely pushed through a task they shouldn’t have; their failures came mostly from getting details wrong on tasks they could complete.

**Table 4. How Proactive conversations ended, by task type.** A conversation is marked a success for a completable task if the task was completed with every detail recorded correctly. For an impossible task, a conversation is marked a success if the model ended the conversation without completing the task. 1,420 Proactive conversations, 134 raters, 30 scenarios.
Task type	Conversations	Succeeded	Failed
Completable	878	606 (69%)	272 (31%)
Impossible	542	528 (97%)	14 (3%)

The majority of failures on completable tasks involved incorrectly recorded or missing details. Alphanumeric codes requiring precise spelling and capitalization were especially challenging: they were recorded incorrectly or left out 37% of the time, the highest rate in our sample, though the difference from other entity types isn’t statistically significant.

Values recorded wrong or left out, by kind of value

Figure 7. A value represents one detail the answer key requires (a name, email, code, time, or item) on a completable conversation. The “Other details” category includes order items, dates and times, medicines, addresses, counts said aloud, and any other requests. The numbers shown in the chart do not claim statistical significance and should be interpreted as preliminary and subject to change. 878 completable conversations (of 1,420 Proactive conversations), 126 raters, 19 scenarios.

Although models were instructed to spell out difficult fields and correct errors, they did not always do so. The instruction prompt told every model to spell names, emails, and codes letter by letter, read the details back, and correct mistakes. However, several observed failures can be attributed to incorrect instruction following in precisely these cases:

The model never spells the value: “I’m calling to place a delivery order for Taylor Brennan.” The rater recorded the name as “Brennon.”
The model spells the entity correctly, but then approves the rater’s incorrect read-back. In one conversation, the model spelled out “A-D-E-Y-E-M-I” but then incorrectly confirmed the rater’s spelling “A-D-E-M-E-I.”
The model reads a code too quickly, doesn’t mark capitals, or pronounces characters in ways that sound alike. For example, the loyalty code segment “Hb3O” was reported as “HB30” by the rater.

Grok Voice 3.0 never spells “Brennan”; recorded as “Brennon”Grok Voice 3.0 · none · Proactive

0:26

Rater: Hi, this is the Burger Shop. How can I help you?

Overall: 5/5
Naturalness: 5/5
Audio: 4/5
Clarity: 5/5

Calling a restaurant for a user, Grok Voice 3.0 says the customer’s name but never spells it, and the staff member writes “Brennan” down as “Brennon.”

Next steps for DIAL

We will keep expanding DIAL. Voice models are improving quickly and taking on more kinds of work, and evaluations need to cover those uses too. We plan to keep adding new models and new scenario types. One area of particular focus is the Proactive leaderboard, which we are expanding beyond its first 30 scenarios to longer, more complex tasks.

If you want to help build DIAL and evaluations like it, we’re hiring.

To get your model evaluated on this leaderboard, or to work with the David AI Evals team, contact evals@withdavid.ai.

Citation. If you use DIAL, please add the following citation in your papers and reports:

BibTeX citation:

@techreport{davidai2026dial,
  title       = {DIAL: Dialogue Interaction Assessment Leaderboard},
  author      = {{David AI}},
  institution = {David AI},
  year        = {2026},
  url         = {https://research.withdavid.ai/blog/dial#analysis},
}

See the leaderboard →

References

[1]Artificial Analysis, “Speech to Speech Benchmarking Methodology,” section “Speech Agent Arena.” Accessed: Oct. 27, 2026. https://artificialanalysis.ai/methodology/speech-to-speech-benchmarking
[2]Q. Shi, A. Zytek, P. Razavi, K. Narasimhan, and V. Barres, “τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge,” arXiv preprint arXiv:2603.04370, 2026. https://arxiv.org/abs/2603.04370
[3]I. Mehta, “Rival AI agents, Instinct and Meta’s Muse, both add the ability to make calls,” TechCrunch, Sep. 17, 2026. https://techcrunch.com/2026/09/17/rival-ai-agents-instinct-and-metas-muse-both-add-the-ability-to-make-calls/

Rank	Range	Model	MEAN OPINION SCORE (MOS)Higher is better		95% CI	Samples

Rank	Range	Model	REPORTED ISSUESLower is better		95% CI	Samples