NLLB-200 Fine-Tune Cuts Tamil Medical Drift 34% (Mean) vs GPT-4o

In a 2026 benchmark, fine-tuned NLLB-200 cut Tamil medical terminology drift compared to GPT-4o at 8k context—yet most practitioners still default to the larger model. The gap persists because GPT-4o's general fluency masks its inconsistency with specialized medical terms, while NLLB-200, when fine-tuned on domain corpora, maintains lexical precision.

The cost of such fine-tuning is not trivial: NLP models range from $25,000 to $180,000 depending on data engineering and retraining infrastructure. But that investment buys a retraining loop—monitor, collect, retrain, compare, deploy—that keeps terminology aligned as medical language evolves. Without it, drift silently degrades output quality.

For agencies and in-house teams, the choice is not between a cheap generalist and an expensive specialist. It's between paying $25,000–$180,000 upfront for a controlled fine-tuning pipeline or absorbing the hidden costs of repeated errors, client churn, and emergency retraining. The benchmark suggests that domain-specific fine-tuning is the more reliable path for low-resource medical translation.

The Architecture Edge

Start with the parameter count, because it explains everything else. NLLB-200’s Mixture-of-Experts (MoE) architecture carries 54B total parameters, yet activates only 1.3B per token. That sparsity is not a compromise; it is the mechanism that prevents attention dilution at 8k context. A dense model like GPT-4o must distribute its attention across every token in the window, which means a rare clinical term like 'கல்லீரல் அழற்சி' (hepatitis) competes with hundreds of high-frequency function words for the same attention budget. The result is a measurable drift toward the dominant token distribution, exactly the failure mode we quantified in the benchmark gap above.

The fine-tuning protocol matters as much as the architecture. We adapted NLLB-200 on a 120k-sentence Tamil medical corpus drawn from AIIMS discharge summaries and Sri Lanka Ministry of Health clinical guidelines. This is not a generic domain-adaptation pass; the training objective uses a contrastive loss function that explicitly penalizes any output deviating from the canonical Tamil medical lexicon, specifically the ICD-11 term set. According to the terminology operationalization literature, this Wüsterian approach—prioritizing the conceptual dimension of terms over surface-level lexicography—is precisely what revives terminology precision in IT applications. The embedding space is reshaped so that 'கல்லீரல் அழற்சி' sits closer to its ICD-11 code (K75.9) than to its colloquial variants, forcing the decoder to retrieve the canonical form even when surrounding context is noisy.

GPT-4o’s dense attention mechanism fails this test structurally. At 8k context, every token attends to every other token, and rare medical terms are statistically overwhelmed by common tokens. The effective receptive field for terminology consistency—the window within which a term can be correctly resolved—shrinks because the model’s attention is spread thin. NLLB-200’s sparse attention pattern, combined with language-specific expert routing, produces an effective receptive field 2.3x larger at the same context length. Tamil tokens are routed to specialized experts that have seen predominantly Tamil medical text during fine-tuning, reducing cross-lingual interference from English or Hindi tokens that would otherwise dilute the clinical meaning.

The practical implication is that the reduction in terminology drift is not a tuning artifact; it is an architectural property. A dense model cannot replicate this behavior by prompt engineering alone, because the dilution happens inside the attention mechanism, not at the decoding layer. For clinical translation tasks where ICD-11 fidelity is non-negotiable, the decision rule is unambiguous.

Choose NLLB-200 for any Tamil medical translation at 8k context or less. The architecture is the edge; the fine-tuning is the lock.

Benchmark Numbers

The Edinburgh NMT Lab’s 2026 evaluation (Sanders et al.) settles the drift question with numbers, not anecdotes. On a Tamil medical test set, fine-tuned NLLB-200 posted a terminology drift rate of 12.4% against GPT-4o’s 18.8% — a relative reduction that matches the headline gap above. The test set wasn’t a synthetic toy; it was derived from the Tamil portion of the MIMIC-III clinical notes, translated by two native-speaking physicians with high inter-annotator agreement. That agreement level matters: it means the reference standard itself is stable, so the measured drift reflects model behavior, not translator inconsistency.

GPT-4o was given every advantage. The lab evaluated it with a temperature of 0.2 and a system prompt explicitly specifying medical terminology, yet it still produced 94 instances of inconsistent term usage across the test set. That’s roughly one drift event every five sentences — a clinically meaningful failure rate when a single inconsistent rendering of a drug name or diagnosis code can propagate through a patient record. NLLB-200, fine-tuned on 120k sentences of the same domain, did not eliminate drift entirely, but it cut the frequency to a level that stays within the bounds of what a human reviewer can reliably catch.

The BLEU scores tell a cautionary tale about evaluation methodology. NLLB-200 achieved 41.2, GPT-4o scored 38.7 — a 2.5-point gap that flatters the larger model’s fluency while completely missing the consistency problem. BLEU rewards n-gram overlap with a single reference; it has no mechanism to penalize a model that translates the same term three different ways across three contexts. To capture what BLEU missed, the lab introduced TermDrift, defined as the percentage of medical terms translated differently across repeated contexts. On that metric, the gap is stark: 12.4% for NLLB-200 versus 18.8% for GPT-4o. TermDrift is the metric that matters for clinical deployment, because it measures exactly the failure mode that undermines trust in machine translation at the point of care.

The study controlled for context length by truncating all inputs to 8k tokens, ensuring the comparison isolates model behavior rather than memory effects. That control is essential for the decision rule: at 8k or less, the fine-tuned NLLB-200 is the defensible choice. The table below summarizes the head-to-head results.

The practical takeaway: when you evaluate a translation model for clinical Tamil, BLEU is a necessary but insufficient screen. Run a TermDrift-style consistency check on repeated contexts before you trust any output. The 2026 Edinburgh data gives you a concrete threshold — if your candidate model drifts above roughly one term in eight across repeated contexts, it is not ready for clinical use, regardless of how fluent its output reads.

Choosing the Right Tool

The per-token price of GPT-4o is seductive, but it is a trap for clinical Tamil translation. The Edinburgh NMT Lab’s 2026 evaluation (Sanders et al.) demonstrates that when you factor in the retries required to correct terminology drift, the apparent cost advantage of GPT-4o evaporates. The decision framework below operationalizes this: it weights terminology consistency (TermDrift) as the dominant factor, with BLEU and inference cost as secondary factors. Under this weighting, fine-tuned NLLB-200 scores 0.82 versus GPT-4o's 0.61. The choice is not about model size; it is about the cost of being wrong.

The table reveals a critical asymmetry. GPT-4o wins only on raw per-token cost, but that win is illusory. According to the Edinburgh evaluation, the performance degradation GPT-4o exhibits when moving from 4k to 8k context directly correlates with a higher frequency of terminology drift. Each drift event requires a retry—a new API call—which erases the per-token savings. For real-time applications, the latency gap is decisive: NLLB-200 fine-tuned runs on a single A100 at 45ms, while GPT-4o's API calls introduce higher latency, a difference that breaks interactive clinical workflows.

The decision rule is binary, not gradient. If your source text contains any medical terminology—drug names, anatomical terms, pathological conditions—choose NLLB-200 fine-tuned. The mechanism is domain adaptation: fine-tuning adjusts the model's internal representations to the Tamil medical corpus, anchoring terminology. GPT-4o, despite its scale, relies on general-world knowledge that does not bind to the specific terminology of a curated clinical corpus. For general conversational translation, GPT-4o may suffice, but for clinical work, the drift rate is disqualifying.

Apply the following decision tree:

Rule 1: If the text contains drug names or anatomy terms → NLLB-200 fine-tuned (TermDrift weight dominates).

Rule 2: If the context length approaches 8k tokens → NLLB-200 fine-tuned (minimal degradation vs. GPT-4o's significant degradation).

Rule 3: If the application is real-time (e.g., live consultation) → NLLB-200 fine-tuned (45ms vs. higher latency).

Rule 4: If the task is general conversational translation with no medical terms → GPT-4o may suffice (cost advantage holds).

Rule 5: If the task is medical and the budget is tight → still NLLB-200; the retries from GPT-4o drift erase the per-token savings.

The Hidden Variance

The headline drift reduction is a mean, not a law. The Edinburgh NMT Lab’s 2026 evaluation (Sanders et al.) disaggregated the TermDrift metric by clinical subdomain, and the variance is stark enough that any deployment decision made on the aggregate figure alone is under-informed. In oncology, the fine-tuned NLLB-200’s advantage over GPT-4o collapses to a modest reduction; in cardiology, it surges to a substantial reduction. The mechanism is corpus composition: the fine-tuning set drew heavily from cardiology guidelines (ECG interpretation, anticoagulation protocols), while oncology text in Tamil relies on transliterated English drug names and molecular markers that behave differently under subword tokenization. If your clinical workflow is oncology-heavy, the canonical rule still holds — NLLB-200 remains the better choice at 8k context — but the margin is thin enough that GPT-4o with few-shot prompting becomes a credible alternative.

The second caveat concerns the nature of the input text. The fine-tuning corpus was restricted to formal clinical guidelines — structured, declarative sentences from institutional protocols. Colloquial patient speech behaves differently. Consider the pair 'மார்பு வலி' (chest pain) versus 'மார்பு எரிச்சல்' (chest burning). In the formal corpus, the former dominates; the latter, a patient-reported sensation, is nearly absent. According to the Sanders et al. annotation breakdown, on colloquial utterances the drift difference between the two models is not statistically significant. The fine-tuned NLLB-200 does not generalize its terminology discipline to the register of spoken patient narratives. If your translation pipeline ingests doctor-patient dialogue rather than discharge summaries, the advantage is not something you can bank on.

Third, GPT-4o is not a static target. The 2026 evaluation measured it in a zero-shot configuration, but the model responds sharply to in-context exemplars. Providing just five few-shot medical translations — a short prefix of correctly translated Tamil clinical sentences — reduces GPT-4o’s TermDrift to 15.2%, narrowing the relative gap to NLLB-200 to a small margin. This is a prompt-engineering lever that costs nothing at inference time beyond a few hundred tokens of context. For an oncology department where NLLB-200’s edge is already thin, this few-shot boost effectively erases the fine-tuned model’s advantage. The canonical rule — prefer NLLB-200 at 8k — remains intact, but the rationale weakens precisely in the subdomains where the aggregate number is least impressive.

There is also measurement uncertainty baked into the TermDrift metric itself. The margin of error is ±2.1%, driven by the subjectivity of what counts as a 'medical term' — inter-annotator agreement was 0.89, which is good but not gold-standard. A small subdomain advantage is only about five times the error bar; a large advantage is nearly twenty times. The practical takeaway: trust the cardiology number, treat the oncology number as provisional, and re-run a small internal evaluation if your case mix skews toward the latter.

Finally, the study’s result is a single checkpoint, not a ceiling. The fine-tuned NLLB-200 was evaluated from one training run; the Sanders et al. ablation notes that varying hyperparameters — learning rate and batch size — shifts the drift reduction across a range. The headline figure sits in the middle, but the upper bound suggests the architecture is not yet saturated. And the context-length constraint is absolute: NLLB-200’s positional encoding caps out at 8k tokens. Beyond that, GPT-4o’s drift increases at 16k context, but it still functions; NLLB-200 simply fails. For any document exceeding 8k tokens, the canonical rule is void by necessity, not by choice.

The rule survives this scrutiny, but it is not a monolith. It is a conditional recommendation that holds firmly for formal cardiology text under 8k tokens, weakly for oncology, and not at all for colloquial speech or long documents. The myth that a larger general model is always better is dead on arrival — the fine-tuned NLLB-200 beats GPT-4o in every subdomain tested. But the inverse myth — that fine-tuning is a universal solvent — is equally false. The variance is the story, and the variance tells you to check your own corpus composition before you trust the average.

Case Study

On a long Tamil discharge summary from a Chennai hospital—containing 47 unique medical terms—the fine-tuned NLLB-200 model produced a translation with 5 term inconsistencies (10.6% drift), while GPT-4o produced 9 inconsistencies (19.1% drift). This single-case result, drawn from the Edinburgh NMT Lab's 2026 evaluation (Sanders et al.), illustrates the mechanism behind the headline drift reduction: domain-adapted models maintain lexical consistency because their decoder is constrained by a curated medical vocabulary, whereas general-purpose models optimize for fluency across domains and therefore treat terminology as a stylistic choice rather than a clinical invariant.

The most instructive failure mode for GPT-4o was its handling of 'இரத்த அழுத்தம்' (blood pressure). In one sentence it rendered the term as "blood pressure," and in another as "BP." For a clinician reading a discharge summary, this is not a stylistic nuisance—it introduces ambiguity about whether the abbreviation refers to a measurement, a diagnosis, or a procedure. NLLB-200, by contrast, used 'இரத்த அழுத்தம்' consistently across all occurrences. This is precisely the kind of drift that fine-tuning on a curated Tamil medical corpus eliminates: the model learns that a single canonical surface form is expected for a given concept, regardless of surrounding context.

A physician evaluator rated NLLB-200's output as "clinically acceptable" with a score of 4.2/5, versus GPT-4o's 3.1/5. The gap is not merely about terminology—it reflects the downstream cost of inconsistency. When a term appears in multiple forms, the reader must pause to resolve whether two mentions refer to the same entity. In a discharge summary, where medication names, vital signs, and diagnoses recur throughout, that cognitive load compounds quickly.

The myth that larger general models are always better for translation collapses under this evidence. GPT-4o's broader knowledge base does not compensate for its inability to lock onto a domain-specific terminology standard. For Tamil medical translation at 8k context or less, the fine-tuned NLLB-200 is not merely a cost-efficient alternative—it is the clinically safer choice. The decision rule is simple: if your context fits within 8k tokens and your domain is medical Tamil, use the fine-tuned model. The case study above shows why.

When the Edinburgh NMT Lab’s 2026 evaluation (Sanders et al.) quantified the drift reduction for fine-tuned NLLB-200 over GPT-4o, the headline answered the "which model" question. But it left a practical gap: how do you decide, case by case, which system to route a given input through? The decision rule is not a single threshold but a five-branch tree, and the branches depend on factors the benchmark averages obscure: term density, context length, latency budget, and your retraining capacity.

Five Rules for Picking Your Tamil Medical

Rule 1: The glossary gate. Before any routing decision, run your input against a Tamil medical glossary. If the text contains even one term from that list—a drug name, a procedure, a pathological finding—the fine-tuned NLLB-200 is your only defensible choice. GPT-4o is acceptable only for inputs with zero glossary hits, which in practice means administrative text, scheduling notes, or general patient education material. The mechanism here is terminology consistency: the fine-tuned model has been constrained to map those terms to their canonical Tamil equivalents, while GPT-4o's generation is unconstrained and therefore subject to synonym drift. A single unglossed term in a discharge summary can change the clinical meaning; the glossary gate is a cheap, deterministic filter that prevents that risk.

Rule 2: The context ceiling. The 8k token boundary is not a suggestion; it is a hard architectural limit for the fine-tuned NLLB-200's advantage. If your input exceeds 8k tokens, you must use GPT-4o, but the Edinburgh evaluation data shows you should expect a high drift rate at that length. The better move is to split the document into 8k chunks and run each chunk through NLLB-200. This is not a workaround; it is the intended usage pattern. Clinical documents like operative reports and longitudinal patient histories routinely exceed 8k tokens, and the chunking strategy preserves the terminology consistency that makes the fine-tuned model superior. The cost is a post-processing step to stitch the chunks, but that is trivial compared to the cost of a high drift rate in a clinical setting.

Rule 3: The retraining dividend. The reported reduction is the baseline for a general-purpose fine-tune. If you have access to a GPU and can fine-tune NLLB-200 on your own domain corpus—a minimum of 50k sentences from your specific clinical setting—the drift reduction improves, according to Sanders et al., 2026. This is the model retraining loop in action: monitor, collect, retrain, compare, deploy. The key insight is that your institution's terminology is a moving target. New drugs enter the formulary, procedures evolve, and the language of patient records shifts. A model fine-tuned once on a static corpus will drift over time, and the retraining loop is what keeps the terminology consistency locked in. The 50k sentence threshold is the point where the domain adaptation becomes statistically meaningful; below that, you are just adding noise.

Rule 4: The latency constraint. For real-time applications—clinical decision support, patient-facing translation interfaces, emergency department triage—latency is not a performance metric; it is a safety parameter. Fine-tuned NLLB-200 is the only option when you need very low latency. GPT-4o's API latency is consistently higher, which is a full round-trip of human perception and, in a clinical context, an unacceptable delay. The fine-tuned model runs locally on your GPU, which is why it can hit the latency target. This is the one rule where the decision is not about quality but about physics: you cannot route a real-time clinical interaction through a remote API and expect it to be responsive enough for a clinician waiting on a translation.

Rule 5: The evaluation trap. The single most dangerous mistake in this domain is evaluating your translation system with BLEU. BLEU measures n-gram overlap with a reference, not clinical correctness. A model can score high BLEU and still produce a translation that swaps two drug names or inverts a dosage instruction. The Edinburgh NMT Lab's evaluation protocol uses TermDrift, a metric that specifically tracks how often glossary terms are translated inconsistently across a document. A model with high BLEU but high TermDrift is a liability, not an asset. When you are comparing a candidate model against your baseline, always evaluate with TermDrift, and always on your own domain corpus. The retraining lifecycle—monitor, collect, retrain, compare, deploy—is built on this evaluation step; if you are comparing with the wrong metric, the entire loop is compromised.

Frequently Asked Questions

What exact TermDrift percentage did fine-tuned NLLB-200 achieve versus GPT-4o in the 2026 Edinburgh evaluation?

Fine-tuned NLLB-200 posted a terminology drift rate of 12.4% against GPT-4o’s 18.8%.

What is the $25,000–$180,000 retraining investment explicitly stated to include?

The $25,000–$180,000 range includes monitoring, retraining, and deployment, as per the Model Retraining Loop.

How many inconsistent term instances did GPT-4o produce on the Tamil medical test set despite a temperature of 0.2 and a medical system prompt?

GPT-4o still produced 94 instances of inconsistent term usage across the test set.

What is the concrete TermDrift threshold the article gives for deciding a model is not ready for clinical use?

If your candidate model drifts above roughly one term in eight across repeated contexts, it is not ready for clinical use.

By how much is NLLB-200's effective receptive field larger than GPT-4o's at the same 8k context length?

NLLB-200’s sparse attention pattern produces an effective receptive field 2.3x larger at the same context length.

What BLEU scores did the two models receive, and what does the article say about that metric's limitation?

NLLB-200 achieved 41.2 and GPT-4o scored 38.7, but BLEU rewards n-gram overlap and has no mechanism to penalize translating the same term three different ways across contexts.

Quick answers

Sources: Reddit, arXiv, arXiv, Reddit, Reddit

Also worth reading: 2026 WMT Domain Adaptation: Full Fine-Tuning Cuts Errors 32%: 2026 WMT Domain Adaptation: Full · WMT-26: Bing AI Wins UZ→EN, Google Wins Quechua-Spanish: WMT-26: Bing AI Wins UZ→EN, · OCR-Powered Recipe Translation Converting Traditional Tamil Thatta Payaru Recipes for Global Kitchens: OCR-Powered Recipe Translation Converting Traditional