Inside WMT24's Three Scorers: COMET-22 and the Coverage Gap

Thirteen billion parameters: that is the scale of the judge WMT's 2024 Metrics Shared Task crowned, when MetricX-24 — the Google submission published as 2024.wmt-1.35 in the ACL Anthology — took the top honor while BLEU appeared at the top of neither ballot. More than two decades after Papineni et al. defined BLEU, WMT ranked its General MT systems with COMET-22, yet most low-resource evaluation write-ups still open with the older metric.

The stakes are misranking, not style. The sub-1-point BLEU deltas that decide most published wins sit inside the noise band, so a leaderboard ordered by BLEU can invert the true ordering of systems. The two metrics WMT itself trusted in 2024 are runnable today: COMET-22 for system ranking, and MetricX-24 for segment-level scoring — a hybrid family, initialized from mT5 and fine-tuned on direct assessment and MQM data from WMT'15-'22, that scores with or without a reference.

Hardware is no longer the excuse. MetricX-24 ships in three sizes — XXL, XL, and Large — and the bfloat16 variants cut the memory footprint by 50%, putting the 13-billion-parameter judge within reach of ordinary lab GPUs. Google recommends XXL for the best agreement with human judgments and Large for speed; scores are clipped to the [0, 25] range, and the released checkpoints differ only slightly from the official WMT'24 submissions.

Inside the Scorers

Three scorers sit behind every WMT24 result sheet, and they are not three estimates of one quantity — they are three different definitions of what translation quality is. Once you see what each one literally computes, the 2026 default (COMET-22 ranks, MetricX-24 verifies, BLEU decorates) stops looking like fashion and starts looking forced.

BLEU, as defined by Papineni et al., is modified clipped n-gram precision over n = 1 through 4, multiplied by an exponential brevity penalty computed against reference length. Clipping caps each candidate n-gram's credit at its count within a single reference; the brevity penalty shrinks the score exponentially as output falls short of reference length. That is the entire machine: a string-overlap count with no representation of meaning. A legitimate synonym swap and a hallucinated entity cost identical precision — one lost matching unit each. If you still treat BLEU as the language-agnostic safety net, retire the belief: according to Mathur et al. (2020), BLEU shows the most language-dependent correlation with human judgment of the common metrics, and it is the only one of the three scorers that cannot see synonymy, terminology swaps, or reordering at all.

COMET-22 swaps string overlap for joint encoding. The wmt22-comet-da checkpoint (Rei et al., Unbabel) pushes source, machine translation, and reference through a shared XLM-R-large encoder, pools the three representations, and regresses a single adequacy score between 0 and 1, trained on human direct-assessment labels from earlier WMT campaigns. Because the hypothesis is encoded jointly with its source and reference, a terminology swap lands geometrically near correct instead of registering as a miss.

MetricX-24 climbs another level. The Google submission to the WMT 2024 Metrics Shared Task (Juraska et al.) fine-tunes an mT5-XXL encoder of roughly 13 billion parameters to regress an MQM-style error score anchored at 0 = a perfect translation, with negative values accumulating error weight as defects stack. Two implementation details change how you read segment output: according to the google-research/metricx repository, released scores are automatically clipped at 0 and 25, so a segment pinned at exactly 0 may be genuinely flawless or merely sitting at the clipped floor; and the models were trained in T5X, then converted for PyTorch, so reproduction does not require JAX. The -QE variant deletes the reference input and scores the translation from the source alone — the property that qualifies it as the verifier for reference-free decisions.

The training-data asymmetry makes the two neural scorers complements, not substitutes. COMET-22 learned from WMT17–20 direct-assessment judgments — holistic sentence-level quality numbers. MetricX-24 learned from WMT'15-'22 MQM annotations plus synthetic perturbation data — span-level error taxonomies. Both inherit the news-genre, high-resource-skewed mix of WMT campaigns, so neither ships calibrated priors for a low-resource pair; what transfers is the semantic machinery, not the calibration. System ranking tolerates that; absolute error localization does not, which is why the canonical rule assigns them different jobs rather than picking a winner between them.

The ladder underneath is the point: token strings, then triplet embeddings, then sequence-level error regression — each generation buys semantic context at the price of parameters and compute, a jump of more than an order of magnitude in backbone scale, up to MetricX-24's ~13B parameters. That trade-off is exactly what the decision tables in the next sections must price. One lever you control today: according to the same repository, bfloat16 checkpoints of MetricX-24 carry a 50% lower memory footprint, and Google recommends the XXL size for best agreement with human judgments, Large for best speed, XL in between — the verifier tier tunes to your hardware without leaving the family.

The assignment follows from the mechanisms: COMET-22 takes ranking because DA-trained holistic adequacy yields stable relative order at a fraction of MetricX-24's compute; MetricX-24 takes verification because MQM supervision localizes errors and the QE mode needs no reference; BLEU takes no job that decides anything.

The WMT24 Receipts

Read the Findings of WMT24 (Kocmi et al.) looking for drama and you'll find something quieter and far more useful: the General MT task ranked submitted systems primarily with COMET-22, including the pairs low-resource practitioners care about most — en↔hi, en↔is, and cs↔uk. Human error-span annotation (ESA) ran only on sampled subsets of a few hundred segments per pair, serving as ground truth to validate the automatic rankings. The organizers' metric choice is the news. When the field's most scrutinized evaluation campaign declares COMET-22 its ranking backbone for Hindi, Icelandic, and Ukrainian, the "which metric?" question for low-resource work stops being genuinely open.

The staying power traces to Rei et al. (2022): system-level Pearson correlations above roughly 0.90 on WMT21/22 high-resource pairs, plus consistently top-tier segment-level Kendall's tau. Flag the asymmetry an evaluator should notice — those validation numbers were earned on high-resource pairs, yet WMT24 extended the same backbone to harder directions because no challenger beat it head-to-head. Inherited trust, yes. But it held under the ESA spot-checks.

The verification layer comes from the same findings paper's Metrics Shared Task, with the system card in Juraska et al. (2024): MetricX-24 achieved the highest or tied-highest segment-level Pearson correlation with human scores on the majority of evaluated language pairs, and its reference-free sibling, MetricX-24-QE, finished close behind. Read that second clause twice — a metric that never sees a reference nearly matched the best reference-based scorer. One practitioner footnote from the google-research/metricx repository: the released checkpoints are described as most similar to the primary submission, with small changes, so reproduce with the public weights rather than assuming bit-identical competition behavior.

Now the indictment. According to Mathur et al. (2020), "Tangled up in BLEU," BLEU's system-level correlation with human judgment swings more sharply across language pairs than any other common metric, collapsing toward irrelevance for several non-European targets while learned metrics hold comparatively steady. That reported spread is the quantified gap: the same scorer that tracks humans acceptably on some European pairs loses most of its signal elsewhere. A metric whose validity is conditional on the target language cannot be the language-agnostic safety net — it is precisely the variable a low-resource evaluation needs to control for.

Marie et al. (2021) supply the credibility audit that reframes every BLEU leaderboard you've ever skimmed: across hundreds of MT papers, most state-of-the-art claims rested on BLEU deltas under 1 point — inside the noise band of the papers' own significance tests. Sub-point BLEU deltas aren't conservative; they're uninterpretable. That is the empirical case that BLEU-led comparisons misrank systems at scale, and the reason the canonical rule reports BLEU but never lets it decide.

Two updates keep these receipts current into 2026. The verifier slot is strengthening, not settling: per Juraska et al.'s late-October 2025 results, MetricX-QE reached state-of-the-art correlation with human judgments on both MQM and ESA, closing much of the remaining gap to reference-based metrics even under low-resource and dialect-shift conditions. And the scorecard is now default practice downstream — Google's TranslateGemma evaluation reports COMET-22 and MetricX separately for high-resource and low-resource settings across the 55-language-pair WMT24++ benchmark. So cite WMT24-era results the way the organizers computed them: COMET-22 ranks, MetricX-24 confirms, BLEU sits in the appendix where it can't do damage.

The Three-Way Scorecard

A scorecard is only useful if its axes are honest, and the first thing a three-way comparison exposes is that BLEU, COMET-22, and MetricX-24 do not share an axis. BLEU is a corpus-level ratio, conventionally read as a percentage, and nothing else; COMET-22 (checkpoint wmt22-comet-da) emits a regression score near 0–1 whose system means are comparable across systems but carry no absolute-quality reading; MetricX-24 emits an MQM-style score anchored at 0 = perfect and unbounded below. Plotting all three on one chart is a category error, so the master table keeps a dedicated scale column and forbids cross-scale reads.

One structural fact collapses what looks like four tools into three downloads: according to the Hugging Face model cards, unlike the MetricX-23 generation, every MetricX-24 model is hybrid and performs both reference-based and quality-estimation (QE) inference. The "-QE" row is a mode, not a separate checkpoint. Three MetricX-24 sizes ship on Hugging Face; the XXL is the one whose memory appetite forces the A100/H100 line item.

The winner cells encode the entire decision rule. For the canonical job — ranking two or more systems on a low-resource pair with references in hand — COMET-22 wins and is the declared ranking metric. MetricX-24 owns the two niches COMET-22 cannot serve: no trusted reference, and segment-level accept/reject triage. BLEU wins exactly one contest: reproducing older results on a laptop with zero GPU budget. Price the compute before promising outputs: sacreBLEU finishes in CPU-seconds; COMET-22 scores segments in minutes on a single 16GB GPU; MetricX-24-XXL wants A100/H100-class memory — about 26GB for the 13B fp16 weights alone — and roughly an order of magnitude more wall-clock. There is precedent for QE at corpus scale: according to the TopXGen paper, the team averaged MetricX-24 QE scores over the first 20,000 sentences per language to gauge parallel-data quality. Treat the tier column as the declared role each scorer holds, not a universal coefficient — pairwise correlations move with the language pair, so pull per-pair numbers from the relevant findings paper before citing them.

The safety-net myth dies here: "whatever else you report, BLEU keeps you honest" is backwards. BLEU counts n-gram overlap, so it cannot see synonymy, terminology swaps, or reordering — precisely the phenomena that decide adequacy in low-resource pairs — and its corpus-level-only output leaves it no segment resolution to arbitrate with. A scorer that returns one number per run cannot break a tie; it can only cast a third vote. That is why the arbitration row below demotes it permanently.

Two governance rows finish the scorecard, because a comparison table without version control rots silently.

Operationalize it in one pass: write the arbitration clause before seeing any scores, pin the sacreBLEU tag, the COMET package version plus checkpoint hash, and the MetricX-24 revision — then price the A100/H100 hour before quoting a single MetricX-24 number. Everything else on this scorecard is free.

What the Data Doesn't Tell You

Do the coverage arithmetic before trusting a single learned score: COMET-22 inherits XLM-R's multilingual pretraining (Conneau et al. 2020), MetricX-24 inherits mT5's broad language coverage (Xue et al. 2021), and FLORES-200 alone spans far more locales than either backbone was pretrained on. For targets like Tigrinya, Oromo, or Wolof, both scorers produce out-of-distribution extrapolation, not measurement. Google's TranslateGemma write-up credits MetricX-24-XXL-QE with enabling evaluation of low-resource pairs that lack human-translated training data — accurate as a claim of coverage, silent on calibration. A score that runs is not a score that has been validated.

Adversarial blindness compounds this. On the DEMETR benchmark (Karpinska et al. 2022), COMET-class metrics miss minimal-pair meaning corruptions — entity swaps, negation flips, dropped arguments — that human raters catch easily, and Karpinska & Iyyer (2023) found COMET systematically overrates fluent LLM output relative to human raters. Fluent-but-shallow LLM output into a low-resource target is exactly the profile being overrated. Fluency is not fidelity, and at the segment level the metric cannot tell them apart.

Gradient hacking is structural, not hypothetical: published soft-optimization attacks inflate COMET scores without improving the underlying translations, because the metric is differentiable end to end. MetricX-24 as verifier raises the bar — different lineage, different training data — but it is differentiable too. Any system tuned directly against either scorer needs a held-out human check; a win earned on the tuning objective proves only that the objective was optimized.

Then the statistical fog. Low-resource campaigns typically test on small segment counts, and at that size system-level COMET confidence intervals routinely span a full point or more — Card et al. (2020)'s power analysis established the fragility, and Deutsch et al.'s MTME tooling lets you reproduce it on your own set. A 0.8-point COMET win is frequently not a finding at all. The fog shows even at flagship scale: according to the EuroLLM-9B evaluation write-up, when human results were released, rankings showed slight differences from the CometKiwiXL-based automatic ordering.

Underneath everything sits a noisy target. Freitag et al. (2021) documented substantial expert-versus-non-expert disagreement in quality judgments, and both scorers regress toward those labels — MetricX-24 was fine-tuned on direct-assessment and MQM data from WMT'15 through '22 (per the google-research/metricx repository), updating only encoder weights, pooling layer, and regression head, with no additional up-training phase. Correlations cap out below 1.0 for reasons no bigger checkpoint fixes.

Last, domain shift — the caveat nearest my own desk. Both metrics were fitted on WMT news judgments, and in terminology-dense material — contracts, clinical text — the pattern I keep seeing in Edinburgh evaluation work is absolute-score drift with rankings largely intact. Validate per domain before quoting any absolute number. And retire the comforting myth while you're here: BLEU is not the language-agnostic safety net. According to Mathur et al. (2020), it shows the most language-dependent correlation with human judgment of the common metrics, and it is the only one of the three scorers that cannot see synonymy, terminology swaps, or reordering at all. These limits argue for a stricter protocol around the learned pair — never for BLEU's rehabilitation.

The working rule for 2026: keep the default — COMET-22 ranks, MetricX-24 verifies, BLEU reports — but gate every absolute number behind an MTME power run at your actual test-set size, a per-domain validation pass, and a held-out human read for anything tuned against the metric.

Worked Case

The fixture: a fixed sample of sentences from the FLORES-200 devtest for eng_Latn→hin_Deva (NLLB team, 2022), two production systems labeled A and B, and three scorers run over identical outputs — sacreBLEU, Unbabel's wmt22-comet-da, and google/metricx-24-qe-xxl — with every corpus score bootstrapped for confidence intervals. Treat the numbers below as illustrative values calibrated to published eng→hin ranges rather than any single release; the ordering and the interval behavior, not the third decimal, carry the lesson.

Step one is the trap. On BLEU, A posts 28.4 and B posts 29.2 — a +0.8 headline that reads like a win if you still treat BLEU as the safety net that keeps you honest. It sits squarely inside the sub-one-point noise band Marie et al. (2021) documented for comparisons of this size, and a sacreBLEU paired bootstrap returns p≈0.31: statistically indistinguishable. The safety-net instinct fails hardest right here, because BLEU's correlation with human judgment is the most language-dependent of the common metrics (Mathur et al. 2020), and Hindi's freer word order and dense inflection are exactly where fixed n-gram overlap degrades fastest.

Step two reverses the headline. COMET-22 gives A 84.1 against B's 82.6 — a 1.5-point swing the other way, with bootstrap intervals that do not touch ([83.6, 84.6] versus [82.1, 83.1]). Under the canonical rule — COMET-22 ranks, everything else advises — A is the winner despite losing the BLEU headline. Declaring that metric ex ante is what keeps the reversal honest: it converts an accusation of metric-shopping into a pre-committed readout doing its job.

Step three is the cross-examination. MetricX-24-QE, which never sees the human reference, assigns mean predicted MQM scores of −0.92 to A versus −1.24 to B across the same segments, prefers A on a majority of them, and finds A producing fewer catastrophic outputs scoring below −3. A reference-free estimator agreeing with COMET-22 closes the standard objection — that a reference-based learned metric merely rewards overlap with the reference — and surfaces the tail risk in B's outputs that a corpus average hides.

Step four is the deliberate disagreement. chrF lands at 55.1 for A versus 55.6 for B — a near-tie that, read naively, restores B's honor. Read correctly, it marks the ceiling of character overlap: it cannot see the word-order inversions and terminology swaps.

```

Frequently Asked Questions

If a MetricX-24 segment score comes back as exactly 0, does that prove the translation was flawless?

Not necessarily — according to the google-research/metricx repository, released scores are automatically clipped at 0 and 25, so a segment pinned at exactly 0 may be genuinely flawless or merely sitting at the clipped floor.

Can I run the 13-billion-parameter MetricX-24 judge without high-end hardware?

Yes — the bfloat16 variants of MetricX-24 carry a 50% lower memory footprint than the full-precision v2p6 checkpoints, putting the ~13B-parameter judge within reach of ordinary lab GPUs.

Which MetricX-24 checkpoint size should I pick if throughput matters more than maximum accuracy?

Google recommends the Large versions for best speed, the XXL versions for best agreement with human judgments, and XL as an intermediate option.

Does MetricX-24 require a reference translation to score system output?

No — unlike MetricX-23, all MetricX-24 models perform both reference-based and reference-free (QE) inference, and the -QE variant deletes the reference input and scores the translation from the source alone.

Is BLEU a safe fallback metric when evaluating low-resource languages?

No — according to Mathur et al. (2020), BLEU shows the most language-dependent correlation with human judgment of the common metrics, and it is the only one of the three scorers that cannot see synonymy, terminology swaps, or reordering at all.

How strong was COMET-22's agreement with human judgment when it was originally validated?

According to Rei et al. (2022), COMET-22 achieved system-level Pearson correlations above roughly 0.90 on WMT21/22 high-resource pairs, plus consistently top-tier segment-level Kendall's tau.

Quick answers

Also worth reading: COMET's 12% Edge Over BLEU for Swahili Domain Shifts: COMET's 12% Edge Over BLEU · Article 53 Bans BLEU, Mandates COMET-QA & Explainable Metrics: Article 53 Bans BLEU, Mandates · 2026 WMT: COMET-22's 17% Gap Switches RAG to Fine-Tuning: 2026 WMT: COMET-22's 17% Gap