MQM vs COMET: which translation evaluation method should you use for AI translations?

The Direct Answer: MQM and COMET Measure Different Things

If you are deciding between MQM (Multidimensional Quality Metrics) and COMET for evaluating machine translation output, the honest answer is that this is not an either-or decision — the two approaches operate at fundamentally different layers of the quality stack. MQM is a human-centric error taxonomy, originally developed at the University of Mainz under Arle Lommel and colleagues and later formalized through work with TAUS and adopted by ISO 18587 as the reference framework for post-editing standards. It asks trained human raters to identify, categorize, and score specific errors in a translation across dimensions such as accuracy, fluency, terminology, style, locale conventions, and design. COMET, by contrast, is a neural reference-based automatic metric introduced by Unbabel researchers in 2020 (Rei et al., published at EMNLP), built on multilingual pretrained models like XLM-R. It predicts how close a machine translation is to a human reference by embedding source, hypothesis, and reference sentences and comparing them in learned vector space.

Also worth reading: How can interpretable translation quality prediction improve AI translations? · How Are Explainable Machine Translation Assessment Methods Shaping AI Translations? · How Does AI Translations Online Compare With Other Translation Tools in 2026?

The practical distinction matters enormously depending on your goal. If you need diagnostic information — knowing why a translation failed, whether it mistranslated a legal term, dropped a negation, or used the wrong register — MQM is the only option that gives you that granularity. If you need fast, cheap, repeatable scoring across thousands or millions of segments during model development or production monitoring, COMET is dramatically more efficient. A single human MQM review of one segment might take two to five minutes for a trained linguist; COMET scores millions of segments per hour on commodity GPU infrastructure at effectively negligible marginal cost. Most mature localization programs in 2026 run both: COMET for continuous automated monitoring and MQM sampling for calibration, root-cause analysis, and vendor accountability.

How MQM Actually Works in Practice

MQM is not a single rigid rubric but a hierarchical taxonomy that organizations can tailor. At its top level sit major categories — Accuracy (subdivided into mistranslation, addition, omission, untranslated text), Fluency (grammar, spelling, punctuation, capitalization), Terminology, Style, Locale Conventions, and Design/Formatting. Raters assign each detected error a severity: typically Critical, Major, and Minor, with Critical errors (such as safety-critical mistranslations or content that could cause legal harm) often defined so severely that a single occurrence fails the segment regardless of other quality. Scoring usually follows a penalty-point model: for example, minus 1 point for a minor error, minus 5 for a major, minus 25 or outright failure for a critical, normalized against word count to produce an MQM score out of 100.

The strengths of this approach are real. Because errors are categorized, you can aggregate findings into actionable reports: if 40% of penalties in a batch come from terminology inconsistencies, the fix is a glossary update, not a model swap. This diagnostic power is why ISO 17100-certified agencies and enterprise buyers like Microsoft, Amazon, and large regulated-industry buyers anchor their quality agreements in MQM-style scoring. The weaknesses are equally real. Inter-annotator agreement between trained human raters on the same content historically hovers around 60–75% correlation, meaning two qualified reviewers can produce materially different scores on identical text. Rater fatigue degrades consistency after roughly 2–3 hours of continuous review. And cost scales linearly: a full MQM review of a 50,000-word documentation set can consume 100–200 professional hours, translating into thousands of dollars before any rework begins.

How COMET Works and What Its Scores Mean

COMET takes a completely different path. Rather than counting discrete errors, it trains a regression model on human judgment data — most famously the WMT (Conference on Machine Translation) Direct Assessment datasets, where humans scored translations from 0 to 100. The current generation, COMET-22 and the newer AfriCOMET and CometKiwi variants released through 2023–2025, uses XLM-R-large encoders fine-tuned to predict human-like adequacy scores. A COMET score near 1.0 indicates very high similarity to the reference; scores are typically reported in ranges like 0.80–0.95 for modern neural MT systems on high-resource language pairs. Unlike BLEU, which matched n-grams literally and correlated poorly with human judgment (often below 0.5 Pearson correlation on WMT data), COMET achieves segment-level correlations with human judgments frequently exceeding 0.85–0.90 on WMT benchmarks.

Two variants matter for different jobs. Reference-based COMET requires a human reference translation, making it ideal for benchmarking MT engines during procurement or model selection when references exist. Reference-free QE variants such as CometKiwi (an uncertainty-aware estimator) score a translation against only the source text, enabling quality estimation in live production pipelines where no reference exists — flagging segments below a threshold (commonly 0.70–0.80 depending on language pair) for human review. This routing function is arguably COMET's highest-value commercial application today: instead of reviewing 100% of machine-translated content, teams route only the bottom 10–20% of segments to linguists, cutting review costs by half or more while maintaining quality gates.

The limitations deserve equal attention. COMET is a black box — it tells you a segment scored 0.72 but not whether the problem is a dropped negation or awkward phrasing. It inherits biases from its training data, performing less reliably on low-resource languages, highly creative text, and domain-specific terminology. Recent research published through Nature and covered by outlets like Slator has examined exactly this gap: studies evaluating literary translation by large language models, including multidimensional assessments of Shen Congwen's Border Town, found that automatic metrics correlate noticeably worse with expert judgments on literary prose than on technical or news text, because metaphor, voice, and cultural resonance resist embedding-space similarity measurement.

Head-to-Head Comparison Table

The strongest programs layer the two methods deliberately rather than choosing one. Start by defining your quality thresholds per content type: a common configuration assigns target MQM scores of 98+ for legal and medical content, 95+ for marketing, and 90+ for user-generated or internal content, while setting COMET QE floor thresholds per language pair based on historical calibration runs. Next, calibrate: take a representative sample of 500–1,000 segments, have qualified linguists perform full MQM analysis, and simultaneously run COMET and CometKiwi on the same segments. Plot the correlation. In many commercial language pairs you will find segments scoring below roughly 0.75 on CometKiwi account for a disproportionate share of MQM critical and major errors — often capturing 70–90% of them while representing only 15–25% of volume. That ratio becomes your routing rule.

Then operationalize. Run COMET-based QE on every segment entering your pipeline, auto-publishing content above threshold and queueing below-threshold segments for human review. Sample 2–5% of all published content for periodic full MQM audits to verify the thresholds still hold, recalibrating quarterly or whenever you change MT engines, since a new engine version shifts score distributions and invalidates old cutoffs. Document severity definitions explicitly — what counts as Critical versus Major varies between buyers, and ambiguity here is the leading cause of disputes with language service providers. Finally, feed MQM error categories back into prompt engineering or fine-tuning cycles: if omissions dominate your Accuracy penalties, that points to a specific, fixable model behavior rather than generic 'low quality.'

Common Mistakes Teams Make

The most frequent error is treating COMET scores as absolute quality verdicts comparable across language pairs, domains, or model versions. A 0.85 in English-to-German technical documentation does not mean the same thing as a 0.85 in English-to-Japanese marketing copy; score distributions differ, and thresholds must be calibrated per pair and per domain. Teams also routinely compare COMET scores produced by different metric versions — COMET-20, COMET-22, and later releases are not interchangeable, and mixing them corrupts trend lines. Pin your metric version the way you pin software dependencies.

On the MQM side, the classic failure is under-specified severity guidelines combined with uncalibrated raters. Without calibration sessions using gold-standard annotated samples, rater drift inflates or deflates scores by several points over time, silently changing what 'pass' means. Another mistake is applying full MQM rigor uniformly to all content, burning budget on low-stakes text where a lightweight QE gate would suffice — and conversely, trusting pure automatic metrics for regulated content where a single undetected critical error carries compliance consequences far exceeding any review savings. Finally, some teams chase metric leaderboard numbers during engine selection without validating on their own data; WMT benchmark wins do not always transfer to specialized domains, as the literary-translation studies noted above demonstrate clearly.

When to Act and What It Costs

If you are publishing machine-translated content today with no systematic evaluation, the priority order is clear: implement reference-free QE scoring first (CometKiwi or equivalent), because it immediately gives you risk-based routing at minimal cost; add MQM sampling second, once volumes justify the spend. Open-source implementations of both exist — COMET models are available via Hugging Face under permissive licenses, and MQM rubrics are publicly documented — so pilot costs are mostly linguist time. Budget expectations: a calibration study of 1,000 dual-scored segments typically runs $2,000–$6,000 depending on language pairs and rates; ongoing MQM auditing at 2–5% sampling adds roughly 1–4% to total localization spend, while QE-driven routing commonly reduces full-review costs by 40–60%. For teams using managed platforms, services such as those offered by AI Translations integrate automatic quality estimation directly into delivery workflows, letting smaller teams adopt this hybrid discipline without building infrastructure themselves. The right moment to act is before a quality incident, not after — retrofitting evaluation methodology onto an angry customer escalation costs multiples of proactive implementation.

Where the Field Is Heading

The boundary between the two paradigms is blurring. Systems like TransEvalnia, covered by MarkTechPost, use prompting-based LLM pipelines to produce fine-grained, human-aligned evaluations that attempt MQM-style error identification at automatic-metric scale — essentially asking large language models to act as MQM annotators. Early results show promising correlation with human raters on some language pairs, though consistency, hallucinated error detection, and cost per segment remain open problems. Meanwhile, hybrid intelligent assessment models combining BERT-based encoders with classifiers such as SVMs, described in recent Nature-published education research, point toward learned error categorization rather than pure regression scoring. Expect the 2026–2028 period to deliver tools that output both a COMET-like score and an MQM-like error breakdown in a single pass. Until those mature and are validated on your specific content, the disciplined pairing of human MQM audits with automated COMET monitoring remains the defensible standard — not because either method is sufficient alone, but because each covers precisely the blind spot of the other.