Can Large Audio Language Models reliably judge speech-to-speech models?

There are three main ways S2S model outputs are scored today, each with its own tradeoffs.

Deterministic metrics compute a repeatable, well-defined score from signal-level metrics on the output (e.g., DSP metrics for acoustic quality, speaker-similarity cosine distance) or a check that the execution environment reached the expected end state.

Model-as-judge evaluation asks another LLM or LALM to rate response quality, either pointwise (one response scored in isolation) or pairwise (relative quality of two responses scored against each other), either from the transcript (LLM-as-judge) or directly from the audio (LALM-as-judge).

Human evaluation asks people to rate model outputs, pointwise or pairwise, in the same settings as above.

Deterministic metrics should be used wherever possible: they’re fast, cheap, interpretable, and comparable across models and evaluation runs over time. When they fail to capture an important quality signal, a model-as-judge approach is a scalable alternative to human ratings, provided you can validate that it correlates well with human preference. Where it doesn’t, you need actual human ratings.

The research to date suggests that judge reliability varies dramatically depending on what dimension of quality you’re measuring, how you elicit the judgment, and how much you’ve invested in specializing the judge.

Judges Align Most Closely With Humans on Lexical Content

When the quality signal lives in the words that were said rather than how they were said, judges can be as reliable as humans. The evidence is somewhat mixed on whether you should expect a cascade of ASR → LLM-as-judge to outperform native LALM-as-judge on these lexically-dominated tasks: some papers find that LALMs outperform a cascade when using a relatively weak ASR model, while others find that cascade approaches tend to outperform LALMs. Regardless, most academic literature is anchored on the previous generation of LALMs (Gemini 2.5 Pro and GPT-4o). It is likely that neither approach strictly dominates today, and the true best judge will vary on the specific task. However, given the token density of text is typically significantly lower than audio, cascade systems are often the best starting point if you have reason to expect quality to be dominated by the lexical content for your evaluation task.

Judges Lag on Naturalness and Paralinguistics

Several studies have found LALM judges to correlate poorly with human judgments on emotion, emphasis, and style. Other studies have found LALMs fail to handle nonverbal dimensions robustly, and are outperformed by cascade systems when the cascaded LLM is provided textual annotations describing the acoustic cues (i.e. once some other system has done the perceptual work).

Specialized Judges Close the Gap, but Don’t Generalize

Recent work has found that fine-tuning an LALM for a specific paralinguistic task can beat frontier generalists LALM models. However, a major downside of this approach is that specialized judges are not general: they are trained and validated on particular distributions of languages, styles, and audio data, and their advantage is measured against benchmarks whose labels carry their own construction choices and annotation noise. Whether and how well a judge fine-tuned on a particular task and data domain transfers to neighboring tasks and domains is not well understood in any theoretical framework today, and would need to be assessed empirically on a case-by-case basis.

Reliability Depends on the Protocol

Even the most capable judges are fragile, with evidence showing that verdicts shift depending on clip ordering, audio concatenation strategy, and overall response length. Pairwise judgments routinely flip under equivalent prompt templates or resampling. Similarly, multiple studies have shown that pairwise preference is steadier than pointwise metrics, but judges still force winners on negligible gaps.

We ran a series of experiments comparing LALM judges against our own calibrated human raters, assessing the output quality of two SOTA S2S models (GPT-Realtime-2, Marin voice; and Gemini 3.1 Flash Live, Kore voice) across 15 rubric dimensions. Model output was captured for 80 human-recorded input prompts, which we split into 40 output pairs for human MOS (Mean Opinion Score, pointwise) assessment and 40 output pairs for human CMOS (Comparative Mean Opinion Score, pairwise) assessment. All 80 pairs were also scored by three LALM judges (Gemini 2.5 Flash, Gemini 2.5 Pro, and GPT-audio). The full protocol is available in the appendix of this post.

Inter-Rater Agreement

Before assessing LALM-as-judge performance, we need to understand inter-rater agreement in our dataset, as this sets the maximum achievable ceiling for a judge model - even a perfect LALM judge cannot show strong correlation with human preferences when humans don’t agree with each other.

To assess the achievable ceiling for an LALM judge model, we first look at split-half correlation across human raters (a randomly selected half of raters correlated against the other half of human raters) by rubric category across MOS scores:

A fuller comparison with CMOS inter-rater agreement metrics, along with an exploration of some alternative correlation metrics, is in the appendix at the end of this post.

Judges Track Human Performance on Content, and Diverge on Acoustics

Consistent with the available literature, the strongest pattern we saw was that judge reliability varies sharply across evaluation dimensions. All three LALM judges agreed closely with human raters on “content” rubrics such as answer quality, relevance, and instruction following. That agreement broke down on acoustic rubrics, including naturalness, emotional appropriateness, and pronunciation.

Judges Split on Acoustic Preference

While judges largely agree with each other on content quality metrics, they have systematic preferences on acoustic quality metrics. In other words, it’s not just that judges are poorly correlated with human ratings - each model also has unique and strong biases. Perhaps unsurprisingly, judges tend to prefer their own family of models on acoustic dimensions:

In principle you would always prefer to use a judge model that’s unrelated to any of the models under test, but in practice there are a limited number of model providers with SOTA performance, and it is not unusual for evals to rely on judge models from the same family as some of the models being evaluated.

Note that systematic bias can exist in the presence of strong correlation; it’s entirely possible for a judge to be correlated (tending to move up and down in sync) with human ratings and also biased (showing a strong preference in one direction on average). Both systematic bias and poor correlation are problematic when relying on models as judges.

LALMs Show (Some) Expected Systematic Biases

Our data largely replicated both the positional and verbosity biases present in the literature, though it is inconsistent across models.

Position bias. GPT-audio showed strong first-clip bias (β = +0.74). The Gemini judges were near-neutral (+0.07 and +0.08). GPT-audio’s bias peaked on the acoustic dimensions where it correlated weakly with humans (naturalness +1.53, speaker drift +1.46, pronunciation +1.03), which looks like a model defaulting to the first option when it’s unsure.

Verbosity bias. LALM judges leaned toward longer responses; humans didn’t. Human CMOS showed no length preference (r = -0.09), whereas the LALM judges all leaned positive (gpt-audio +0.20, gemini-flash +0.23, gemini-pro +0.03). This is important because GPT-Realtime-2 tends to produce longer responses than Gemini 3.1 Flash Live (+7.9s mean, longer 70% of the time). The pattern held even among the 147 ratings where humans called a tie (3-judge average +0.16), splitting to +0.35 when GPT-Realtime-2 was the longer response and -0.18 when Gemini 3.1 Flash Live was. The consistent direction across three judges, against a flat human baseline, suggests verbosity bias is a judge-side artifact rather than a genuine quality signal.

Human and LALM Judges Fail Differently

Human failures are mostly operational - clicking the wrong button, losing focus, fatigue, and anchoring on distractor quality cues like loudness. These failures can be mitigated through a combination of scale, calibration, and proper experimental design (volume normalization, randomized order, blind review, agreement monitoring, etc.).

LALM judge failures are characteristically different - hallucination, overconfidence, sycophancy, and reasoning breakdowns. Here is a rollout from our dataset that makes this point clear:

We made this prompt intentionally vague to test whether a model asks for clarification instead of hallucinating. Humans strongly preferred the Gemini response for answer quality, while LALM judges strongly preferred the GPT-Realtime-2 response that hallucinated details not in the prompt. Investigating the judge reasoning traces, we see they invented a shared visual context (assuming the user could see it) and rewarded the model that hallucinated detail. All three penalized the clarifying question as a non-answer.

This particular failure mode could likely be patched with careful prompt engineering of the judge model - spelling out that it should judge responses only on the provided details and treat extraneous details as a hallucination - but that only works for failure modes you already know about. Identifying the “unknown unknown” failure modes means manually reviewing model output, starting with the cases where human and judge preferences diverge most sharply.

Aggregate Metrics Hide Real Failures

Even a well-designed metric can bury critical failures that it’s not designed to surface - a metric that weights every sample equally (like MOS and CMOS) will bury a rare but severe failure among dozens of ordinary ones. If the catastrophic failure rate is what you care about, an averaged preference score simply won’t surface that, and you need a separate pass built to catch the tail.

Likewise, some problems are obvious to a human listener but may not map cleanly onto any single rubric, so other factors drown them out even on the rubric meant to capture them. Below are a few examples of this phenomenon from our evaluation:

Listening note: these audio defects are noticeable, but can be easy to miss in a noisy environment. We recommend listening with headphones or in a quiet, sound-isolated setting.

Scores are only part of the evaluation signal. While pairwise judgments are generally considered more sensitive and reliable than pointwise judgments, we found that raters consistently wrote sharper free-text comments on the MOS tasks than on CMOS. We hypothesize that judging a sample in isolation pushes people to attend to the sample itself rather than the comparison. Whatever the cause, those comments were among the most useful signals we collected, and indicate that running a single MOS pass with a free-text field purely for the qualitative layer may be valuable, even when CMOS is your headline metric.

Limitations

These are early results, not a full academic paper, and we want to be clear-eyed about the limitations in this analysis:

• Easy prompts. We built the prompt set to sit well within reach of modern S2S models, so the high human scores are unsurprising. The human-versus-judge gap could look quite different on harder or more divisive prompts, where answers grow more complex and inter-rater agreement drops.
• Small sample. The analysis rests on 40 prompts through MOS and 40 through CMOS. The patterns are suggestive rather than definitive, and we would want more data before treating any single number as settled.
• Fixed rubrics. We did not experiment with rubric wording. These may not be the right dimensions, and sharper phrasing could shift the results for humans and judges alike.
• Judge-model overlap. The judges come from the same families as the systems under evaluation. That is hard to avoid with so few SOTA providers, but it remains a real confound. We tried open source judges as a neutral alternative, but they trailed badly on agreement with humans.
• Judge-model quality. We did not evaluate Gemini 3.1 Pro Preview or GPT-Audio-1.5 as judge models because of concerns that they may share the same base model or training mix as the models under evaluation.

Any general conclusions about LALM-as-judge performance should be tempered by the understanding that current-gen models are likely the best judges as well as the best-performing S2S models.