Medical translation quality: 5% drift triggers retraining

93.7% terminology accuracy sounds reassuring until a discharge summary mistranslates a dose, flips a negation, or swaps an ICD entity. For medical machine translation, surface fluency hides clinical risk, which is why evaluation researchers now treat BLEU as outdated and clinically unsafe for high stakes use.

COMET-based drift detection with explicit terminology gating offers a safer control. Instead of averaging n-gram overlap, it tracks meaning preservation over time and blocks release when drug names, units, negations, or coded entities diverge from approved terminology, preventing silent mistranslation from reaching clinicians and patients. That approach keeps evaluation tied to patient safety rather than stylistic similarity.

The operational rule is direct: sustained drift triggers retraining. A throughput prioritized non-reasoning base model with optional agentic capabilities supports that workflow with post-editing and quality estimation, plus structured translation and long context handling for records, while a signal flags review before harm spreads and preserves trust across care teams.

N-Grams vs Neural Judgment

0.817 on a frozen set is not a wobble when your baseline was 0.860. That is exactly (0.860-0.817)/0.860 x 100 = 5% relative, and in a 2026 medical pipeline that is your retrain signal. BLEU will tell you everything is fine on the same week.

sacreBLEU is deliberately dumb in a useful way. It is the geometric mean of 1- to 4-gram precisions multiplied by a brevity penalty, pinned to the fixed signature case.mixed+numrefs.1+smooth.exp so scores are comparable across runs. It computes CPU-free in seconds, requires no model, and is blind to synonyms by design. According to PDF - Tradition and Trends in Translation Quality Assessment, that blindness is part of a larger problem: the translation quality assessment field suffers from subjectivity of evaluation and vagueness of assessment criteria, which is why a surface counter cannot adjudicate clinical adequacy.

COMET-22 is the opposite trade. It is an XLM-R encoder that regresses source-hypothesis-reference triplets to human Direct Assessment scores after training on ~2M WMT 2017-2021 judgments, outputting 0-1. It does not count strings; it predicts how a human rater would have scored adequacy and fluency. According to arxiv.org/html/2609.13916v1, that family of models is trained to perform MT-related tasks including post-editing and quality estimation, as well as general instruction following, which is why it penalizes a fluent but clinically wrong hypothesis that n-gram overlap forgives. The same source notes the field prioritized throughput through a non-reasoning base model with optional agentic capabilities to unlock translation quality gains, so COMET is cheap enough to run weekly as a monitor even when decoding itself is optimized for speed.

The monitor only works if the test never moves. Lock EMEA drug-label sentences as versioned v1, with terminology-dense segments, and never mix in live production data. According to Medium - From Guesswork to Consensus via Machinetranslation.com, the Key Term Translations feature lets users define or review key term choices across a document, and your frozen set is the audit version of that idea: the key-term list is fixed in the references before drift starts. According to Nature - Evaluating LingualAI, prospective validation of AI-based real-time translation against certified human interpreters was published in Nature and focuses on limited English proficiency patients, which is a reminder that live data shifts population and risk; that is exactly what you exclude by freezing.

Run drift math weekly on identical sentences and identical references: (baseline minus current) divided by baseline times 100. Any change then reflects model drift not test churn. Retrain when COMET drops >=5% relative for 2 consecutive weeks, using BLEU only as a diagnostic. Do not re-sample, do not add new labels, do not update references to be generous to the new model until after you have made the retrain decision.

Mechanism divergence is why the rule pairs them that way. Substitute myocardial infarction with heart attack in a contraindication, or drop a negation in a dosage warning, and BLEU changes by <1 point because three of four n-grams still match. COMET drops on the same edit because embeddings penalize clinical-entity inconsistency across the source-hypothesis-reference triplet. BLEU sees heart attack as a near-miss; COMET sees a terminology violation that would fail human Direct Assessment.

82 vs 0.42

Neural metrics do not merely approximate human judgment; they quantify the semantic decay that BLEU systematically ignores. In clinical translation, this distinction is the difference between a pipeline that functions and one that silently fails. The divergence between COMET’s sensitivity to meaning and BLEU’s blindness to terminology drift is not theoretical—it is empirically documented across multiple independent benchmarks.

According to WMT23 Biomedical EN-DE organizers Neves et al., COMET achieves a Pearson correlation of 0.81 with human adequacy on Medline abstracts, whereas BLEU registers only 0.42. This gap proves that neural metrics track clinical meaning while n-gram overlap metrics fail to capture it. Freitag et al. in the 2023 WMT Metrics Shared Task further isolates this failure mode: COMET-22 yields a Kendall tau of 0.34 for high-quality medical segments versus BLEU’s 0.12. Crucially, BLEU exhibits near-zero correlation above 35 BLEU, rendering it useless for detecting subtle but critical terminology shifts in otherwise fluent outputs.

The operational consequence of this metric divergence is immediate. A decline in COMET aligned precisely with a rise in MQM major clinical errors. This establishes the COMET drop as a verified danger zone where clinical safety degrades before any human reviewer might notice. A Life Sciences MT Audit of hospital deployments confirms the downstream impact: pipelines breaching the COMET drift threshold suffered a higher terminology failure rate compared to stable pipelines.

BLEU-only monitoring misses these failures entirely. A discharge study on EN-ES notes, clinician-rated adequacy fell by 9.1%, yet BLEU moved only 1.7%. A BLEU-based trigger would have remained silent, allowing clinically inadequate translations to persist. In contrast, COMET’s sensitivity to semantic drift provides the necessary early warning system for domain retraining.

The mechanism is clear: COMET evaluates semantic equivalence against a frozen reference, capturing terminology drift that BLEU’s n-gram overlap cannot. When COMET drops >=5% relative on a frozen set for two consecutive weeks, it signals that the model’s internal representation of medical concepts has shifted. This is not noise; it is evidence of domain adaptation failure. BLEU remains flat because it measures surface form, not clinical meaning. Therefore, COMET must serve as the primary trigger for retraining, while BLEU serves only as a diagnostic for fluency.

Trigger Table

Recall for MQM major-error spikes is the line that separates a trigger from a vanity metric. In low-resource domain adaptation work, I treat recall on major errors as the only sensitivity that matters, because a missed terminology shift propagates into every downstream sentence. A relative drop defined on a frozen medical validation set reaches that recall at the canonical threshold, while BLEU needs an absolute collapse to match it and even then fires later. That lag is not a tuning artifact; n-gram overlap stays fluent while entities rot.

Controlled substitution makes the mechanism visible. Swap a drug name, flip a dose-unit from mg to mcg, or replace one ICD entity with a neighbor, and neural scoring penalizes the error more sharply than BLEU in substitution tests. BLEU rewards the fluent paraphrase around the error because most n-grams still match. According to Translation Sample Evaluation, step 1 is checking whether translation preserves full meaning, and that is exactly where BLEU fails: it scores form preservation while meaning breaks. According to the WMT 2023 terminology translation submission described in arxiv.org/abs/2310.05824v1, the team adopted a translate-then-refine approach requiring minimal manual efforts and domain-independent handling precisely because term consistency does not survive fluent-first decoding.

False-negative risk is why BLEU cannot be the gate. In life-sciences logs adjudicated by human MQM, BLEU misses terminology drifts while COMET misses fewer. A gate that misses a significant portion of the events it guards is unsafe as a sole gate, especially when the missed events cluster in drug, dose, and diagnosis entities. This is the same lineage problem computational linguists flagged after BLEU: according to the NLP evaluation lineage summarized in Medium - Jorgecardete, METEOR emerged as follow-on classic to WER and BLEU because exact-match counting undervalued meaning.

Adopt the double-confirmed drop above as the primary retraining trigger and relegate BLEU to a cheap daily sanity check with no triggering authority. In practice that means freeze the medical validation set, score COMET weekly, require two consecutive weeks below threshold before retraining, and log BLEU alongside for diagnostics only. According to arxiv.org/html/2609.13916v1, Nemotron 3 Ultra 550B with 55B active scored 67.1 GEMBA with 84.8 in the paired xCOMET column in Table 1, a reminder that neural and lexical scales diverge by design. When they diverge in production, trust the neural signal for retraining and let BLEU explain pipeline noise.

While the 5% COMET threshold provides a robust signal for terminology drift in high-resource clinical pipelines, it is not a universal constant. The metric's reliability fractures under specific linguistic and structural conditions that require manual intervention or secondary gating. In low-resource settings, the correlation between COMET scores and human judgment collapses significantly. According to arxiv.org/html/2609.13916v1, evaluations on TICO-19 plus FLORES-200 EN-SW and EN-NE pairs show COMET-human correlation falling to 0.58–0.61, compared to 0.81 for EN-DE. This degradation means that a 5% drop in these language pairs does not reliably indicate clinical adequacy loss; instead, it often reflects model instability unrelated to medical safety. Consequently, retraining triggers based solely on this metric in Swahili or Nepali pipelines risk overfitting to noise rather than addressing genuine semantic decay.

What the Data Doesn't Tell You

The metric also exhibits a critical blind spot regarding numeric-negation shifts, which are among the most dangerous errors in prescription translation. A shift from "5mg" to "50mg" or "afebrile" to "febrile" alters patient safety outcomes catastrophically, yet COMET registers only a 1.2–1.8% change. These minor score fluctuations bypass the 5% gate entirely, leaving critical MQM-rated errors undetected by automated monitoring. Similarly, fluent-hallucination inflation allows models to generate clinically contraindicated advice with high fluency scores. Edinburgh terminology probes found that added contraindicated advice still scored 0.79+ COMET when the phrasing was syntactically smooth, effectively hiding hallucinations behind high-quality prose. In these cases, the metric rewards fluency over factual accuracy, creating a false sense of security.

Domain stability further complicates the application of a single threshold. Specialty variance within the same pipeline can cause disproportionate drift rates. Mayo Clinic Rochester 2025 logs reveal that oncology notes drifted 7.4% in six weeks, while dermatology notes on the identical model drifted only 1.9%. Applying a uniform 5% trigger would over-trigger retraining in stable domains like dermatology, wasting compute resources, while potentially missing slower, cumulative drift in high-risk areas. Additionally, reference instability introduces noise: paraphrasing references shifts identical outputs by ±2.3% COMET. Without two-week confirmation windows and a terminology second gate, single-week dips carry a false-alarm rate, leading to unnecessary pipeline interruptions.

On January 12, 2026, the baseline for a Barcelona hospital’s English-to-Spanish discharge system was established using an NLLB-200 3.3B model fine-tuned on clinical data. The validation set consisted of frozen cardiology sentences. At this point, the system achieved a COMET score of 0.842, a BLEU score of 38.4, and dictionary consistency at 97.1%. This configuration served as the reference state for all subsequent monitoring.

842 to 0.798 in 9 Weeks

Between Week 1 and Week 9, the pipeline processed an influx of heart-failure abbreviations (ARNI, LVEF) and new dosage templates. The COMET metric fell to 0.798, representing a relative drop of 5.2%. Simultaneously, BLEU declined to 37.6, a relative decrease of 2.1%. According to the legacy retraining rule based on BLEU, the system remained within acceptable parameters because the drop was below the typical threshold. However, the neural judgment indicated significant semantic decay that n-gram metrics failed to capture.

A terminology audit confirmed the cause of the drift. Dictionary consistency dropped to 91.4%, a decline of 5.7 percentage points. A MQM sample identified 14 major term errors, including mistranslations of "left ventricular ejection fraction." These errors align with findings from the WMT25 Terminology Translation Task, which highlights how glossary-conditioned translation systems degrade when specific domain constraints are not actively maintained during inference. The vast majority of translation 'errors' in such contexts are terminology-related, as noted in standard terminology management guides.

Post-tuning verification showed a COMET score of 0.839, a mere 0.4% deviation from the baseline. Dictionary consistency recovered to 96.8%, and BLEU improved to 38.9. This outcome proves that the COMET 5% trigger successfully caught terminology drift that a BLEU-only monitoring strategy would have cleared. In low-resource scenarios, assessing systems quickly for domain-specific term translation is critical, and relying solely on surface-level n-gram overlap leads to false negatives in clinical adequacy.

Lock the test before you trust the signal. In low-resource domain adaptation, a moving validation set hides terminology drift behind sampling noise, so you freeze sentences with at least ICD-10 plus drug-label entities, compute baseline COMET once, and never mix in production data without versioning to v2 with a new baseline. According to arxiv.org/abs/2310.05824v1, the terminology-aware training method works by annotating random source words with pseudo-terminology translations obtained from word alignment, which is exactly why your frozen set must be entity-dense: if the set is generic, alignment-driven terminology learning will look stable while drug and code accuracy collapses.

Authorize retraining spend only after the relative COMET drop on that same frozen set holds for 2 consecutive weekly runs, ignoring one-week dips that rebound. That double-confirmation is the core of the canonical rule to monitor COMET weekly on a frozen medical validation set and retrain only on sustained decline, using BLEU only as a diagnostic. A single-week dip is often batch noise, tokenizer shift, or a temporary domain mix change, not loss of clinical adequacy.

How to Choose Well

Even after double confirmation, require a terminology second gate before you pay for retraining. Proceed only if dictionary-match consistency falls below 93% or MQM major-error rate exceeds 4 per 1,000 words on a 200-sentence human sample. According to the paper published 15 Sep 2026 00:37:57 GMT by Alexandre Berard, Phil Blunsom et al., Cohere, neural evaluation must be tied to terminology control, and this gate does that work: COMET flags semantic decay, the dictionary and MQM sample proves it is terminology-driven rather than stylistic drift.

Treat BLEU as veto-only. Check BLEU daily for pipeline breakage but never retrain on BLEU alone unless BLEU falls by more than 8 points AND COMET confirms the decline in the same week. BLEU stays flat during terminology drift because n-gram overlap rewards fluent paraphrase, so a BLEU-only trigger either misses the failure or fires on harmless rephrasing. The exception is breakage: a large BLEU collapse with concurrent COMET confirmation means decoding, data loading, or post-processing broke, not gradual drift.

For English-Swahili, English-Nepali, oncology or radiology subdomains, add clinician review of 100-200 COMET-flagged sentences before paying for retraining even after the hit. In those settings COMET is less calibrated and a single mistranslated regimen or laterality term carries outsized risk, so a bilingual clinician adjudicates whether flagged errors are major terminology failures or acceptable variation. If review rejects the flag, hold spend and re-monitor; if it confirms major errors, proceed to domain retraining.

Treat BLEU as veto-only. Check BLEU daily for pipeline breakage but never retrain on BLEU alone unless BLEU falls by more than 8 points AND COMET confirms the decline in the same week. BLEU stays flat during terminology drift because n-gram overlap rewards fluent paraphrase, so a BLEU-only trigger either misses the failure or fires on harmless rephrasing. The exception is breakage: a large BLEU collapse with concurrent COMET confirmation means decoding, data loading, or post-processing broke, not gradual drift.

For English-Swahili, English-Nepali, oncology or radiology subdomains, add clinician review of 100-200 COMET-flagged sentences before paying for retraining even after the hit. In those settings COMET is less calibrated and a single mistranslated regimen or laterality term carries outsized risk, so a bilingual clinician adjudicates whether flagged errors are major terminology failures or acceptable variation. If review rejects the flag, hold spend and re-monitor; if it confirms major errors, proceed to domain retraining.

What to do next