# Translation metric results: 2024 WMT neural (COMET) vs n-gram (BLEU): Fine-Tune or Not?

Lauren Sanders · October 13, 2026

> Fine-tune only if COMET improves by ≥2 points on domain test set; BLEU must stay flat or rise to avoid overfitting to n-gram overlap.

| Takeaway | Detail |
| --- | --- |
| Fine-tune only if COMET improves by at least 2 points over base model on domain test set | COMET gain ≥2 points is required for fine-tuning to be justified per thesis |
| BLEU must stay flat or rise when fine-tuning; if only BLEU moves, you overfit to n-gram overlap | BLEU flat or rising is required condition; isolated BLEU gain indicates overfit per thesis |
| Score both base and fine-tuned checkpoints with COMET and chrF on held-out in-domain test set | Reader rule mandates COMET and chrF scoring — not BLEU alone — on in-domain held-out set |
| Terminology consistency must improve when fine-tuning | Reader rule requires terminology consistency improvement as joint condition with COMET gain |

This guide delivers a two-metric decision rule for fine-tuning 2024 WMT-era translation systems.

Apply COMET ≥2-point gain and chrF/terminology checks on in-domain held-out data to avoid overfit.

![Translation metric results](https://static.mm-ais.com/article-images-ai/translation-metric-results-2024-wmt-neur-ai-257699c1.jpg)

## How COMET and BLEU score differ

BLEU measures translation quality by counting how many n-grams in the hypothesis appear in the reference, clipped to avoid overcounting, then combining precision scores for 1- to 4-grams with a brevity penalty that penalizes short outputs. This means BLEU rewards surface-level word overlap and cannot distinguish between a correct synonym swap and a meaningless word change, as both may yield identical n-gram matches.

COMET, in contrast, uses a neural encoder to embed the source sentence, hypothesis, and reference into a shared semantic space, then regresses a quality score toward human judgment collected in WMT evaluations. Because it operates on meaning representations rather than token sequences, COMET assigns higher scores to translations that preserve intent through paraphrasing or terminology-consistent substitutions, even when surface forms differ.

To observe this difference empirically, take 20 hypothesis-reference pairs from your domain test set where a single term has been replaced with a valid synonym (e.g., "car" → "automobile" in an automotive context). Run both BLEU and COMET on these pairs. You will typically see BLEU scores remain nearly unchanged or drop slightly due to disrupted n-gram matches, while COMET scores increase or remain stable because the meaning is preserved.

This mechanism explains why COMET can detect improvements in terminology consistency or fluency that BLEU misses: BLEU is blind to semantic equivalence beyond exact n-gram matches, whereas COMET’s neural architecture learns to align with human assessments of adequacy and fluency. Thus, a COMET gain of ≥2 points signals meaningful progress in translation quality, while a BLEU-only increase may reflect superficial overfitting to reference phrasing.

For practical validation, compare the two metrics on your fine-tuned model’s output against the base model using this synonym-swap test set. If COMET shows the expected gain without BLEU degrading, the fine-tuning has likely improved semantic fidelity rather than merely memorizing n-gram patterns from the reference.

![How COMET and BLEU score differ — Translation metric results](https://static.mm-ais.com/article-images-ai/translation-metric-results-2024-wmt-neur-ai-a3713516.jpg)

## The evidence: what sources converge

WMT’s own metric-tracking work shows that COMET-based neural metrics became the primary ranking signal precisely because BLEU and chrF failed to align with human adequacy judgments on morphologically rich and low-resource language pairs. This divergence is documented in the WMT metrics findings papers, where human rankings consistently favored translations scored higher by COMET even when n-gram metrics were flat or lower — a pattern especially pronounced in terminology-dense domains like legal, medical, or technical translation for under-resourced language pairs.

MachineTranslation.com (Tomedes) provides a live, real-world demonstration of why single-metric, single-model evaluation is insufficient: the platform aggregates outputs from 22 AI models per request and ranks them using neural metrics like COMET. The observable variance in rankings across models — even for identical input — proves that relying on one model and one metric (e.g., BLEU alone) underdetermines translation quality, particularly when terminology consistency and semantic fidelity are critical.

The convergence finding is this: neural metrics (COMET, chrF++) and n-gram metrics (BLEU) diverge most significantly on terminology-dense, low-resource output — and you can verify this yourself. Run your base and fine-tuned models on a held-out in-domain test set, score both with COMET and chrF (not BLEU alone), and compare. If COMET improves by ≥2 points while chrF does not drop significantly, and terminology consistency (measured via term base recall or glossary hit rate) improves, then fine-tuning has paid off. If only BLEU rises, you are likely overfitting to surface n-gram overlap without gaining semantic or terminological accuracy.

To self-verify the COMET/n-gram divergence in your domain: select a test set with high terminology density (e.g., >15% domain-specific terms from a glossary) and limited parallel data (

Canonical: https://aitranslations.io/blog/translation-metric-results-2024-wmt-neural-comet-vs-n-gram-bleu-fine-tune-or-not.php
Markdown: https://aitranslations.io/blog/translation-metric-results-2024-wmt-neural-comet-vs-n-gram-bleu-fine-tune-or-not.php/index.md