COMET vs BLEU: Which machine translation evaluation metric should I actually use in 2026?

The Short Answer: COMET Has Largely Superseded BLEU for Most Modern Evaluation

If you are evaluating machine translation output in 2026 and you are still relying exclusively on BLEU, you are working with a metric that the research community has effectively retired as a primary evaluation tool. COMET (Crosslingual Optimized Metric for Evaluation of Translation), developed at Unbabel and refined through the WMT shared tasks since 2020, correlates substantially better with human judgment than BLEU does — typically by margins of 0.10 to 0.20 in terms of segment-level Pearson correlation on WMT benchmark data. BLEU, introduced by Papineni and colleagues at IBM back in 2002, measures n-gram overlap between a candidate translation and reference translations. It was a reasonable proxy in 2002 when neural networks were not in the picture, but it has known and well-documented weaknesses: it ignores synonyms, penalizes valid paraphrases, saturates once translations get good, and produces segment-level scores that often bear no relationship to whether an individual sentence is actually acceptable.

Also worth reading: How Can AI Translations Improve RAG Healthcare Translation Evaluation? · Which LLM Translation Evaluation Metrics Deliver Reliable Production Results? · How Do AI Translation Evaluation Methods Measure Quality Across Languages?

That said, the honest answer is more layered than 'COMET wins.' BLEU is still cheap, deterministic, reference-based, and universally understood, which makes it useful for quick regression checks and for comparing checkpoints within the same system on the same test set. COMET is more expensive to run (it requires a GPU and a neural model checkpoint), its scores depend on which checkpoint you use, and newer LLM-based evaluators are beginning to challenge COMET itself. The Ghent University MATEO project, which works on improving access to machine translation evaluation, reflects a broader European research push to make these newer metrics more accessible to practitioners rather than locking them inside big-lab workflows. This guide walks through what each metric does, where each one fails, how to choose between them, and what the practical workflow looks like.

How BLEU Actually Works — and Why It Breaks Down

BLEU (Bilingual Evaluation Understudy) computes the geometric mean of modified precision scores for n-grams of length 1 through 4 between the machine output and one or more references, multiplied by a brevity penalty that punishes translations shorter than the reference. Because it is a pure string-matching calculation, it runs in milliseconds on any CPU, requires no model download, and returns the identical score every time. These properties explain why it survived for two decades and why it is still embedded in countless CI pipelines and benchmark leaderboards.

The problems appear as soon as translation quality improves or the language pair gets morphologically rich. First, BLEU only rewards exact lexical matches: a translation that uses 'automobile' where the reference says 'car' earns zero credit for that word. Second, BLEU scores are not interpretable at the segment level. A sentence scoring 0.30 BLEU might be perfectly acceptable while a sentence scoring 0.60 might be unusable, because the score reflects overlap, not adequacy or fluency. Third, BLEU saturates. On high-resource pairs like English–French, top systems now score in the low-to-mid 40s, and the differences between systems have compressed to the point where a 0.5 BLEU gain may be statistically meaningless or may reflect nothing but test-set noise. Fourth, on agglutinative languages — Turkish, Finnish, Hungarian, and many Indic languages like Tamil and Telugu — single words carry what English splits across several words, so n-gram overlap becomes a crude instrument that punishes legitimate morphological variation. Sarvam AI's published evaluation work on Indic automatic speech recognition makes exactly this point: surface-level metrics like WER (the ASR analog of BLEU) systematically misrepresent quality for Indian languages, and the same logic transfers to MT evaluation for those language pairs.

How COMET Works — Neural Evaluation with Learned Quality Judgments

COMET takes a fundamentally different approach: instead of comparing strings, it embeds the source sentence, the machine translation, and the reference into a shared multilingual representation space using a pretrained encoder (XLM-R in the original versions, more recent models in later checkpoints such as the reference-based and reference-free variants released through 2022–2024). It then runs a regression head trained on human judgment data from WMT evaluations — direct assessments, DARR scores, and increasingly MQM annotations — to predict a quality score. Because the model learned from thousands of actual human judgments, it knows that 'car' and 'automobile' mean the same thing, that a reordered clause can be equally correct, and that a hallucinated span of fluent text that ignores the source is a serious error even when it overlaps nicely with the reference.

Concretely, COMET comes in several flavors worth knowing about. The reference-based variants require a human reference translation, just like BLEU. The reference-free (or quality estimation) variants, such as CometKiwi, score a translation against the source alone, which matters enormously in production settings where you have no reference because you are translating real user content. On the WMT metrics shared tasks from 2020 onward, COMET consistently ranked at or near the top for correlation with human judgment, with system-level accuracies in the 90%+ range on many language pairs — numbers BLEU never approached. The cost side matters too: running COMET on a few thousand segments takes seconds on a modern GPU, but evaluating millions of segments or running it inside a continuous integration loop on CPU-only infrastructure is a genuine operational consideration. Free checkpoints are available on Hugging Face, and tools like sacreCOMET or Unbabel's COMET package make usage a matter of a few lines of Python.

Direct Comparison: BLEU vs COMET vs LLM-Based Metrics

The table below summarizes the practical differences you will weigh when choosing an evaluation approach. Note that a third category — LLM-as-a-judge metrics, including GEMBA and related GPT-based scorers — has entered the conversation since 2023 and is worth taking seriously, since fine-tuning LLMs with translation memories has been shown to boost accuracy and cut costs, and those same LLMs can evaluate as well as translate.

One honest caveat: COMET is not a perfect judge. It inherits biases from its training data, it can be gamed by systems tuned against it (a known phenomenon called metric overfitting or 'metric hacking' in WMT system submissions), and checkpoints differ enough that a COMET score from 2021 is not comparable to one from 2024. Anyone reporting COMET scores should always name the exact checkpoint — for example 'wmt22-comet-da' — alongside the number.

Practical Steps: Setting Up a Sensible Evaluation Workflow

A defensible evaluation setup does not require choosing one metric; it requires layering them by purpose. Start with a fixed test set of at least 1,000 segments per language pair, drawn from real content your users actually care about, not just generic news test sets like newstest. Divide it by domain and difficulty if you can. Run your baseline and candidate systems on this set, computing BLEU with sacreBLEU (which fixes tokenization inconsistencies that plagued older BLEU implementations — always report the signature string) and COMET with a pinned checkpoint such as wmt22-comet-da or a newer release, reported with its exact name.

For system-level decisions — 'is model B better than model A by a real margin?' — use COMET plus a significance test such as paired bootstrap resampling; BLEU differences under about 1 point are rarely meaningful. For segment-level triage in production, use a reference-free metric like CometKiwi or an LLM judge to flag translations below a quality threshold for human review; typical thresholds fall in the 0.80–0.85 range on the 0–1 COMET-DA scale, but you should calibrate the cutoff against a few hundred human-reviewed segments from your own data rather than trusting any published number. This tiered approach — neural metric for decisions, string metric for cheap regression, reference-free QE for live traffic — is what most serious translation teams and language-service providers converged on by 2024–2025, and it aligns with the direction of research showcased at WMT and in industry write-ups from outlets like Slator covering evaluation projects such as MATEO.

Common Mistakes People Make with These Metrics

The most frequent error is comparing BLEU scores across different tokenizations, test sets, or tools and treating the comparison as valid. BLEU computed with different tokenizers or on different splits of data can differ by several points for reasons that have nothing to do with translation quality. Always compute both systems' scores under identical conditions, and prefer sacreBLEU for its standardized signature. The second common mistake is treating a single COMET score as an absolute quality rating. A 0.85 from one checkpoint does not equal a 0.85 from another, and COMET scores are not percentages of sentences that are correct — they are relative quality signals calibrated on WMT human-judgment data.

Third, people optimize models against a single metric until the metric stops reflecting reality; this is metric hacking, and it is why WMT periodically refreshes its primary metric. Fourth, on low-resource and morphologically complex languages — much of the Indic language space, African languages, and many others — published benchmarks built on BLEU-era infrastructure understate real quality differences; Sarvam AI's Indic ASR work demonstrates the same failure pattern for speech metrics, and the lesson generalizes. Fifth, teams sometimes skip human evaluation entirely because automated metrics are cheap. No current metric fully replaces human judgment for high-stakes content: medical, legal, and safety-critical text still needs MQM-style human review on sampled segments, ideally 200–500 segments per release cycle. Metrics are for scaling decisions, not for replacing accountability.

When Should You Still Use BLEU, and When Is the Switch Overdue?

There remain three legitimate niches for BLEU in 2026. Use it for ultra-cheap continuous regression testing where you only need to know 'did this build break something obvious?' — a BLEU drop of 3+ points against yesterday's checkpoint is a real alarm even if small differences are noise. Use it when reproducing legacy results or comparing against older papers that only report BLEU. And use it in compute-constrained environments where no GPU is available and the test sets are small. Beyond those cases, if you are making model-selection decisions, reporting benchmark results, comparing vendors, or monitoring live translation quality, the switch to COMET (plus a reference-free QE metric for production) is overdue and costs you little beyond a GPU and an afternoon of setup.

The timing question has a concrete answer tied to the state of the field. The WMT metrics shared task has crowned neural metrics as the primary measure of progress since 2020, and the 2024–2026 period has seen LLM-based evaluators gain ground further. Research out of Ghent University's MATEO project and similar efforts is actively making modern evaluation more accessible, which lowers the practical barrier every year. If your organization is evaluating vendors for a translation rollout, or building an AI translation product where quality regressions cost customers, you should have moved past BLEU-only evaluation no later than now. If you are a hobbyist translating a blog post, none of this matters much — human eyes beat every metric at that scale.

Cost, Infrastructure, and What This Means for AI Translation Services

Budget-wise, the evaluation stack is modest. sacreBLEU is free and runs anywhere. COMET checkpoints on Hugging Face are free under permissive licenses; running 10,000 segments on a single consumer GPU takes on the order of a minute, and cloud GPU time costs roughly $0.30–$2.00 per hour depending on provider and hardware, so even a heavy evaluation regime costs dollars per month. LLM-as-a-judge evaluation is the expensive option: depending on the model and prompt design, costs can run from fractions of a cent to several cents per segment, which becomes material at millions of segments — a reason teams reserve LLM judging for sampled audits rather than full-corpus scoring. Human MQM review remains the priciest layer, typically priced per segment by language-service providers, which is exactly why smart sampling guided by QE scores matters: evaluate everything cheaply, review the risky 5–10% with humans.

For organizations evaluating AI translation services — whether in-house systems or commercial providers — the practical takeaway is to demand modern metrics from vendors. Ask any translation provider to report quality using a named COMET checkpoint and, ideally, a reference-free QE score per segment, alongside any legacy BLEU numbers for continuity. Providers serving low-resource and Indic language pairs should be able to explain how they evaluate beyond string-matching metrics, following the pattern Sarvam AI demonstrated for Indic ASR: surface metrics mislead, and semantic or LLM-based evaluation exposes errors that n-gram overlap hides. A vendor that cannot articulate this is evaluating with tools two decades old, and that gap will show up in your production quality.