Translation metric results: 2024 WMT neural (COMET) vs n-gram (BLEU): Fine-Tune or Not?

This guide delivers a two-metric decision rule for fine-tuning 2024 WMT-era translation systems.

Apply COMET ≥2-point gain and chrF/terminology checks on in-domain held-out data to avoid overfit.

How COMET and BLEU score differ

BLEU measures translation quality by counting how many n-grams in the hypothesis appear in the reference, clipped to avoid overcounting, then combining precision scores for 1- to 4-grams with a brevity penalty that penalizes short outputs. This means BLEU rewards surface-level word overlap and cannot distinguish between a correct synonym swap and a meaningless word change, as both may yield identical n-gram matches.

COMET, in contrast, uses a neural encoder to embed the source sentence, hypothesis, and reference into a shared semantic space, then regresses a quality score toward human judgment collected in WMT evaluations. Because it operates on meaning representations rather than token sequences, COMET assigns higher scores to translations that preserve intent through paraphrasing or terminology-consistent substitutions, even when surface forms differ.

To observe this difference empirically, take 20 hypothesis-reference pairs from your domain test set where a single term has been replaced with a valid synonym (e.g., "car" → "automobile" in an automotive context). Run both BLEU and COMET on these pairs. You will typically see BLEU scores remain nearly unchanged or drop slightly due to disrupted n-gram matches, while COMET scores increase or remain stable because the meaning is preserved.

This mechanism explains why COMET can detect improvements in terminology consistency or fluency that BLEU misses: BLEU is blind to semantic equivalence beyond exact n-gram matches, whereas COMET’s neural architecture learns to align with human assessments of adequacy and fluency. Thus, a COMET gain of ≥2 points signals meaningful progress in translation quality, while a BLEU-only increase may reflect superficial overfitting to reference phrasing.

For practical validation, compare the two metrics on your fine-tuned model’s output against the base model using this synonym-swap test set. If COMET shows the expected gain without BLEU degrading, the fine-tuning has likely improved semantic fidelity rather than merely memorizing n-gram patterns from the reference.

The evidence: what sources converge

WMT’s own metric-tracking work shows that COMET-based neural metrics became the primary ranking signal precisely because BLEU and chrF failed to align with human adequacy judgments on morphologically rich and low-resource language pairs. This divergence is documented in the WMT metrics findings papers, where human rankings consistently favored translations scored higher by COMET even when n-gram metrics were flat or lower — a pattern especially pronounced in terminology-dense domains like legal, medical, or technical translation for under-resourced language pairs.

MachineTranslation.com (Tomedes) provides a live, real-world demonstration of why single-metric, single-model evaluation is insufficient: the platform aggregates outputs from 22 AI models per request and ranks them using neural metrics like COMET. The observable variance in rankings across models — even for identical input — proves that relying on one model and one metric (e.g., BLEU alone) underdetermines translation quality, particularly when terminology consistency and semantic fidelity are critical.

The convergence finding is this: neural metrics (COMET, chrF++) and n-gram metrics (BLEU) diverge most significantly on terminology-dense, low-resource output — and you can verify this yourself. Run your base and fine-tuned models on a held-out in-domain test set, score both with COMET and chrF (not BLEU alone), and compare. If COMET improves by ≥2 points while chrF does not drop significantly, and terminology consistency (measured via term base recall or glossary hit rate) improves, then fine-tuning has paid off. If only BLEU rises, you are likely overfitting to surface n-gram overlap without gaining semantic or terminological accuracy.

To self-verify the COMET/n-gram divergence in your domain: select a test set with high terminology density (e.g., >15% domain-specific terms from a glossary) and limited parallel data (<5,000 sentence pairs). Compute COMET and BLEU for both base and fine-tuned models. In low-resource, terminology-rich settings, you will frequently observe cases where COMET gains ≥2 points (indicating better alignment with human judgment) while BLEU changes by <1 point or even decreases — a direct signal that n-gram metrics are blind to semantic improvements in term translation.

This pattern is not anecdotal: it reflects a known limitation of n-gram metrics in agglutinative, morphologically complex, or low-resource languages where word order and inflection vary widely, making surface n-gram overlap a poor proxy for meaning. COMET, by contrast, uses contextual embeddings trained on human judgment data, making it sensitive to terminological correctness and semantic fidelity even when surface form varies.

Therefore, the rule is clear: fine-tune only when COMET gains ≥2 points AND terminology consistency improves on your in-domain test set. If BLEU moves but COMET does not, distrust the gain — it reflects n-gram memorization, not translation quality. This two-metric gate prevents overfitting and ensures fine-tuning efforts target actual usability in real-world, terminology-sensitive scenarios.

Options compared: base vs fine-tune

The base model—such as an NLLB-class or WMT-winning system—requires zero training cost and serves as the correct default when your in-domain test set contains fewer than 1,000 sentence pairs. It generalizes best to out-of-domain or general text and avoids the risk of catastrophic forgetting that can degrade performance on broader language patterns.

Full fine-tuning on in-domain data offers the highest ceiling for terminology consistency but should only be attempted when you have 5,000 or more in-domain sentence pairs and a reliable held-out set for validation. Below this threshold, the model risks overfitting to limited data and losing general translation ability, making the cost unjustified.

Adapter or LoRA fine-tuning provides a practical middle ground: low training cost, fast convergence, and swappable domain modules. For most practitioners, this approach wins because it preserves the base model’s general knowledge while adapting to in-domain terminology—delivering COMET gains without the forgetting risk of full fine-tuning.

Compare the three options directly: base model (zero cost, safe default), full fine-tune (highest terminology ceiling, requires 5k+ pairs), and adapter/LoRA (low cost, swappable, best balance). The adapter/LoRA approach is the winner for most users because it achieves meaningful COMET improvements (≥2 points) and terminology consistency without exceeding feasible data or compute thresholds—making it the empirically supported choice when scaling adaptation across domains.

Costs and numbers that matter

Run both base and fine-tuned checkpoints on a held-out in-domain test set and score them with COMET and chrF. If the COMET gain is less than 2 points, treat it as noise — typical checkpoint variance exceeds this margin, so fine-tuning is not justified.

For data volume, observe that below approximately 1,000 in-domain sentence pairs, fine-tuning more often reduces COMET than improves it, based on empirical trends in low-resource adaptation. At 5,000+ pairs, full fine-tuning becomes defensible as COMET gains stabilize and exceed the 2-point gate with greater consistency.

Frame compute costs practically: a LoRA adapter fine-tune on a ~1B-parameter MT model runs in hours on a single consumer GPU. If your monthly domain translation volume is under 100k segments, the energy and time cost of fine-tuning likely exceeds the value gained versus using the base model or an API.

When volume exceeds 100k segments/month, amortize the one-time fine-tune cost over the output. At this scale, even a modest COMET gain of 2+ points translates to meaningful quality improvement per segment, making the hours-scale GPU job economically rational.

Always verify terminology consistency alongside COMET — gains in neural metric without improved term alignment indicate superficial fluency, not true domain adaptation. Use chrF as a complementary check, but never let BLEU alone drive the decision.

What the evidence does not establish

For 2024 WMT-era systems, the two-metric gate — requiring ≥2-point COMET gain with flat or rising BLEU — does not universally apply. Its validity depends on test set integrity, domain characteristics, and metric reliability. This section identifies specific edge cases where the gate fails or requires adaptation, based on empirical patterns observed in low-resource and terminology-sensitive translation scenarios.

In reference-free domains — such as emerging technical jargon or dialectal speech with no reliable in-domain test set — the gate breaks because COMET cannot be meaningfully computed without human references. Here, the reader should fall back to chrF scores against a small, hand-built reference set of 50–100 segments, supplemented by periodic human spot-checks for adequacy and fluency. This approach, validated in community-driven translation efforts for under-resourced languages, ensures quality signals are not lost when automated metrics lack grounding.

Metric gaming presents another failure mode: if the fine-tuning data overlaps with the test set at the document level — even if sentences are disjoint — COMET gains may reflect memorization rather than generalization. The gate still holds only when the test set is strictly held out and disjoint by document, ensuring no topical or stylistic leakage. Violating this condition inflates COMET artificially, making gains untrustworthy regardless of BLEU behavior.

For very low-resource language pairs where COMET itself was weakly validated during training — such as those with limited parallel data in WMT’s metric development — the gate breaks due to poor metric calibration. In these cases, COMET scores may not correlate reliably with human judgment. The reader should instead rely on chrF for n-gram overlap and conduct targeted human adequacy evaluations, prioritizing semantic fidelity over fluency when resources for evaluation are constrained.

These edge cases do not invalidate the gate where it applies but define its boundaries of usefulness. When the test set is clean, references exist, and COMET is well-calibrated, the two-point rule remains a robust guardrail against overfitting to n-gram metrics. Deviations from these conditions require procedural adjustments, not abandonment of the principle that fine-tuning must demonstrate measurable, human-aligned improvement.

En→Nepali legal MT

For En→Nepali legal MT, run both base and fine-tuned checkpoints on a held-out in-domain test set and score with COMET and chrF — not BLEU alone — to avoid over-reliance on n-gram overlap. Use the base model’s scores as the reference point for evaluating gains, as demonstrated in the worked example where base COMET was 0.72 and chrF 52.1 (or equivalent scale).

After LoRA fine-tuning, if COMET improves by ≥2 points (e.g., from 0.72 to 0.75 or equivalent), the semantic adequacy gain is validated even if BLEU dips slightly, as n-gram stability alone does not reflect terminology or adequacy improvements in low-resource, morphologically rich outputs. chrF’s modest rise aligns with better morphological handling.

A terminology audit using a domain glossary (e.g., 50 terms) can confirm whether COMET gains stem from better terminology consistency: if the fine-tuned model improves term translation rate (e.g., from 34/50 to 45/50), this supports the COMET signal. chrF’s rise aligns with better subword and morphological handling without over-penalizing length.

Worked Example: Run the Numbers

To test fine-tuning impact, run base and fine-tuned checkpoints on a held-out in-domain test set (e.g., 500 sentence pairs) and score with COMET and chrF — not BLEU alone. For example, a base model scoring COMET 0.72 and fine-tuned model scoring COMET 0.75 shows a 0.03 gain, which equals a 3.0-point increase on a 0–100 scale.

To compute COMET delta, subtract base COMET from fine-tuned COMET. If COMET is on a 0–1 scale, multiply the difference by 100 to get points. A gain of 0.03 (e.g., 0.75 − 0.72) equals 3.0 points, which meets the ≥2-point threshold for considering fine-tuning worthwhile.

Check BLEU stability: if fine-tuned BLEU is equal to or higher than base BLEU (e.g., 28.6 vs 28.4), the change is flat or rising, satisfying the condition that BLEU must not fall to avoid overfit to n-gram overlap.

Illustration 3: Verify terminology consistency. The team manually audits 50 random sentences for key term translation (e.g., "myocardial infarction" → "infarto de miocardio"). Base model: 42/50 correct (84%). Fine-tuned model: 47/50 correct (94%). This 10-point terminology consistency gain supports the COMET signal and confirms the improvement is not spurious.

Declaration: For this example, fine-tuning paid for itself. COMET improved by ≥2 points (3.0 gained), BLEU did not fall (rose 0.2), and terminology consistency improved. The break-even trigger is COMET ≥0.74 (a 2.0-point gain from base 0.72); the actual fine-tuned COMET of 0.75 clears this bar.

Decision rules: fine-tune

If COMET gain on held-out in-domain data is less than 2 points, then do not fine-tune — the delta is indistinguishable from checkpoint noise. This threshold ensures you only proceed when the improvement exceeds typical variation between random seeds or training runs, as observed in WMT-era system stability analyses.

If you have fewer than ~1,000 in-domain parallel pairs, skip training and instead select among existing models. For example, compare outputs across the 22 models exposed by MachineTranslation.com — selection is cheaper than training and avoids overfitting on minimal data, a finding consistent with low-resource MT behavior documented in neural metric divergence studies.

If COMET gains ≥2 points AND glossary consistency improves AND out-of-domain COMET drops ≤1 point, then ship the adapter as a domain-locked module. This condition balances in-domain gain with terminology fidelity and cross-domain safety, ensuring the adaptation does not degrade general translation ability beyond an acceptable margin.

If COMET gains ≥2 points but glossary consistency does not improve, reject the fine-tune — you are overfitting to fluent n-gram overlap rather than meaning preservation. This aligns with the principle that COMET, unlike BLEU, reflects human judgment on adequacy, particularly in terminology-dense contexts where surface metrics fail.

If COMET gains ≥2 points, glossary consistency improves, but out-of-domain COMET drops >1 point, then either reduce adapter strength or revert to base model selection. Excessive domain specialization risks catastrophic forgetting, a trade-off visible when neural metrics diverge sharply from n-gram scores on morphologically rich or low-resource language pairs.

As a shortcut, if you lack resources for fine-tuning, simply select the best-performing base model from the 22 available on MachineTranslation.com using COMET and chrF on your held-out test set — this often yields better ROI than training from scratch on limited data.

What to do next

Frequently Asked Questions

What is the minimum COMET gain required to justify fine-tuning according to the thesis?

COMET gain ≥2 points is required for fine-tuning to be justified per thesis.

Which metrics must be scored on the held-out in-domain test set when evaluating base and fine-tuned checkpoints?

Reader rule mandates COMET and chrF scoring — not BLEU alone — on in-domain held-out set.

What condition must terminology consistency meet when fine-tuning to avoid overfitting?

Terminology consistency must improve when fine-tuning as a joint condition with COMET gain.

What happens if only BLEU improves during fine-tuning while COMET does not gain at least 2 points?

If only BLEU moves, you overfit to n-gram overlap; isolated BLEU gain indicates overfit per thesis.

What must be true about BLEU when fine-tuning is justified under the two-metric decision rule?

BLEU must stay flat or rise when fine-tuning; if only BLEU moves, you overfit to n-gram overlap.

Why is relying solely on BLEU insufficient for evaluating translation quality in this context?

BLEU rewards surface-level word overlap and cannot distinguish between a correct synonym swap and a meaningless word change, as both may yield identical n-gram matches.

Quick answers

Also worth reading: Translation metric comparison: COMET vs. character n-gram F-score (chrF)—2026 primary pick: Translation metric comparison: COMET vs. · 2026 WMT: COMET-22's 17% Gap Switches RAG to Fine-Tuning: 2026 WMT: COMET-22's 17% Gap · COMET-22 Scores, WMT Data, and the 2026 Translation Decision: COMET-22 Scores, WMT Data, and