COMET-QE vs LLM translation evaluation: which quality metric should you trust in 2026?

The Short Answer: Reference-Free Metrics Have Won, But Not Completely

If you are evaluating machine translation output at scale in 2026 and you cannot rely on human reviewers for every segment, the practical answer is this: use COMET-QE (or a comparable reference-free neural metric) as your primary automated gate, and use LLM-as-a-judge evaluation as a secondary layer for error analysis, style assessment, and edge cases where regression-based metrics struggle. Neither tool alone gives you the full picture, and treating either one as a ground truth is the single most common mistake teams make when they build translation quality pipelines.

Also worth reading: What is agent as judge translation and how does it improve upon traditional LLM-as-a-judge evaluation methods? · What are the definitive sovereign translation vendor evaluation criteria for organizations handling sensitive data in 2026? · What is the accurate WES certified translation cost breakdown for immigration and academic evaluation?

The distinction matters because these two approaches measure fundamentally different things. COMET-QE is a reference-free metric derived from the COMET family of models developed at Unbabel and refined through the WMT shared tasks. It scores a translation against its source text alone, predicting adequacy without needing a human reference translation. LLM-based evaluation, by contrast, uses large language models such as GPT-4-class or Claude-class systems to judge translations, often following rubrics like MQM (Multidimensional Quality Metrics) categories of accuracy, fluency, terminology, style, and locale conventions. Research presented around AMTA 2025 and covered by Slator showed that teams combining both approaches — using metrics to triage segments and LLM judges to explain failures — improved their post-editing workflows measurably compared to relying on either method alone.

This article breaks down how each approach works, where each one fails, what the research actually says about correlation with human judgment, and how to build an evaluation stack that costs less than the quality problems it prevents.

How COMET-QE Actually Works Under the Hood

COMET-QE belongs to the QE (quality estimation) branch of the COMET metric family. The original COMET model was trained on human judgments from WMT evaluation campaigns, learning to embed a source sentence, a hypothesis translation, and (for reference-based variants) a reference translation into a shared vector space, then predict a score between roughly 0 and 1 that correlates with human adequacy judgments. The QE variant drops the reference entirely. It compares the source embedding against the hypothesis embedding and predicts how adequate the translation is, making it usable on live production traffic where no reference exists.

Several properties follow from this architecture. First, scores are segment-level and continuous, typically reported on a 0–100 scale after rescaling, so you can set thresholds like "flag anything below 70 for review." Second, the metric is multilingual: modern COMET models cover dozens of language pairs within a single checkpoint, though performance degrades on low-resource languages where training data was thin. Third, it is sensitive to hallucinations — untranslated spans, omissions, and source-detached fluent nonsense — which older metrics like BLEU were notoriously blind to. This hallucination sensitivity is precisely why reference-free neural metrics displaced BLEU and chrF in production monitoring between roughly 2021 and 2024.

There are real limitations worth stating plainly. COMET-QE scores can be inflated by translations that are fluent but subtly wrong in meaning, because embedding similarity captures semantic overlap better than logical precision. Numbers, named entities, negation scope, and cross-sentence discourse errors are known weak spots. The metric also carries a domain bias: a model trained heavily on news and general web text will miscalibrate on legal contracts, clinical trial protocols, or patent claims unless you fine-tune or at least validate on your own domain data. Finally, scores are not comparable across different COMET versions — a 0.85 from one checkpoint does not equal a 0.85 from another — so any pipeline upgrade requires recalibration of your thresholds.

How LLM-Based Translation Evaluation Works

LLM evaluation takes a different route entirely. Instead of a specialized scoring model, you prompt a general-purpose large language model with the source text, the machine translation, optionally a glossary or style guide, and a rubric. The model returns either a score, a set of error annotations mapped to MQM categories, or a free-form critique. Because the underlying model has broad world knowledge and strong instruction-following, LLM judges can catch things embedding-similarity metrics structurally cannot: wrong dates, swapped entities, mistranslated idioms, tone mismatches against brand voice, and violations of client-specific terminology lists supplied in the prompt.

Research presented at AMTA 2025 and reported by Slator highlighted exactly this complementary behavior. Teams found that LLM judges aligned well with human reviewers on error typology — telling you what went wrong — while regression-style metrics like COMET-QE remained faster and cheaper for deciding whether a segment needs attention at all. Several conference talks described hybrid pipelines: run the cheap metric over 100% of segments, send the bottom decile plus a random sample to an LLM judge for annotation, and reserve human post-editing for confirmed problem segments.

The weaknesses here are equally concrete. LLM judges exhibit position bias, verbosity bias, and self-preference bias — they tend to rate outputs from models similar to themselves more favorably. Scores drift across model versions; upgrading your judge model silently changes your quality baseline. Prompt sensitivity is severe: changing a few words in the rubric can shift average scores by several points. And cost per segment runs anywhere from 5 to 50 times higher than a local COMET inference, depending on prompt length and whether you include few-shot examples. Judge outputs also need structured-output enforcement (JSON schemas, constrained decoding) or you will spend engineering time parsing prose instead of measuring quality.

Head-to-Head Comparison

The table makes the strategic picture clear: these tools occupy different points on the cost-precision-explainability trade-off curve, not competing points on the same axis.

What the 2025–2026 Research Actually Says

The honest reading of recent literature, including the AMTA 2025 material Slator summarized, is more cautious than vendor marketing. Three findings stand out. First, no automated metric — neural regression or LLM judge — reliably replaces human judgment on high-stakes content. Correlation coefficients in the 0.8s sound impressive until you realize that means meaningful disagreement on a substantial minority of segments, and disagreement concentrates precisely on the hard, ambiguous cases that matter most commercially. Second, metric performance varies sharply by language pair and domain; headline numbers come disproportionately from English-centric, high-resource pairs like EN→DE and EN→ES, while results on, say, EN→TH or DE→JA legal content are weaker and less consistently reported. Third, the most effective documented use case is not absolute scoring but relative ranking: comparing two MT engine outputs, measuring improvement after a fine-tune, or detecting regressions after a model update. Relative comparisons cancel out much of the calibration noise.

A fourth, subtler finding deserves attention: metrics can be gamed. If you optimize an MT system directly against COMET, you get translations tuned to the metric's blind spots — fluent, semantically adjacent, occasionally factually off. Teams running continuous evaluation loops should periodically spot-check with humans specifically to detect metric overfitting. The same applies to LLM judges, which can be steered toward leniency by models trained on similar instruction data.

A Practical Pipeline You Can Build This Quarter

Start by establishing a golden set: 500 to 2,000 representative segments from your actual content, professionally reviewed by qualified linguists, with MQM error annotations. Without this anchor, every threshold you set is arbitrary. Budget roughly $1,000–$5,000 for professional review depending on volume and language count; this is the highest-ROI spend in the entire evaluation program.

Next, run COMET-QE over the golden set and compute correlation with your human scores per language pair. If Pearson correlation falls below about 0.6 on a given pair, do not trust raw thresholds there — either fine-tune the metric on a few hundred of your annotated segments or fall back to heavier LLM/human sampling for that pair. Set alerting thresholds at the point where human reviewers agree a segment needs editing; for many commercial content types this lands somewhere between 60 and 75 on the rescaled COMET scale, but your golden set decides, not a blog post.

Then layer in LLM judging as a sampled audit. Send the lowest-scoring 5–10% of segments plus a random 1–2% control sample to an LLM judge with your glossary and MQM rubric. Track three numbers weekly: the flag rate, the precision of flags (what fraction of flagged segments humans would also edit), and the escape rate (errors found in the random control sample that the metric scored highly). If escapes exceed roughly 1–2% of the control sample, tighten thresholds or expand the audited fraction. Re-baseline everything whenever you change the MT engine, the COMET checkpoint, or the judge model — version changes invalidate historical comparisons.

Finally, close the loop with humans on a fixed cadence. A monthly review of 200–400 randomly sampled translated segments, scored by two independent linguists with adjudication, keeps every automated layer honest and catches systematic drift before customers do.

Common Mistakes That Invalidate Entire Evaluation Programs

The most frequent failure is threshold cargo-culting: copying a "score below 70 = bad" rule from a vendor deck without validating it on your own data. Domains differ enough that this routinely produces either alarm fatigue (everything flagged, reviewers stop trusting the system) or false comfort (real errors sailing through). The second mistake is ignoring language-pair variance — a pipeline validated on Spanish tells you nothing about Korean. Third is conflating metric versions: upgrading COMET without re-running your golden set makes trend charts meaningless, yet this happens constantly because dashboards keep plotting new scores against old baselines.

Fourth is over-trusting LLM judges because their explanations read persuasively. Fluency of explanation is not accuracy of judgment; studies repeatedly show LLM evaluators miss errors they confidently claim to have checked. Fifth is evaluating only finished translations rather than the full workflow — if your process includes MT output selection among multiple engines, light post-editing, or terminology injection, evaluate the end product, not the intermediate step. Sixth is neglecting the random control sample. Auditing only low-scoring segments guarantees you never learn about the errors your metric cannot see, which are, by definition, the dangerous ones.

When to Act, and What It Costs

If you ship machine-translated content today with no automated quality measurement, the priority order is: build the golden set first (one to two weeks, $1K–$5K), deploy COMET-QE monitoring second (days of engineering if you self-host the open-weight model on a single GPU, or near-zero marginal cost via managed APIs), add LLM-judge auditing third (a week of prompt engineering plus ongoing API spend), and schedule recurring human audits last but permanently. Total initial investment for a mid-size localization program typically lands between $10,000 and $40,000 including engineering time, with ongoing costs dominated by human review — which is exactly as it should be, since human judgment remains the only ground truth available.

Act now if any of these apply: you changed MT engines or fine-tuned a model in the last quarter, you are entering regulated domains (medical, legal, financial) where error tolerance is near zero, your customer-reported error rate has crept upward, or you are being asked to expand into language pairs you have never measured. Delay is defensible only when volumes are tiny and every segment already receives human review — in that case, automate later and spend the budget on the translators.

For teams weighing vendors versus building in-house, note that platforms in the AI translation space — including services like AI Translations — increasingly bundle reference-free quality estimation and reporting alongside the translation itself, which can shortcut the infrastructure work described above. Evaluate those bundled metrics against your own golden set exactly as you would a self-hosted model; the validation discipline does not change based on who hosts the model.

The Bottom Line

COMET-QE and LLM-based evaluation answer different questions, and the question determines the tool. Ask COMET-QE: "Is this segment probably fine?" — millions of times per day, cheaply. Ask an LLM judge: "What specifically is wrong with this segment, and does it violate our terminology and style rules?" — on a targeted sample. Ask human experts: "Is our entire measurement system still telling the truth?" — on a fixed schedule. Teams that respect this division of labor get production-grade quality assurance at a fraction of full-human-review cost. Teams that pick one metric, set an unvalidated threshold, and stop thinking about it will discover the gap the expensive way: through customer complaints.