# NLLB-200 50K Medical Scoreboard: Tune vs Constrained

Lauren Sanders · September 4, 2026

> NLLB-200 50K Medical Scoreboard: Tune vs Constrained. Over one billion words per year across up to 85 languages forces SAP to maintai...

| Takeaway | Detail |
| --- | --- |
| Inline annotation learns terminology in context | Dinu et al. (2019) variant using inline annotation of target terms in source plus source factor embeddings during training and inference |
| Context learning matters where tools are missing | SAP requirement to learn context due to lack of reliable term recognition or morphological inflection generation tools for all productive languages |
| Translate-then-refine catches violations | Alignment to discover whether a terminology constraint has been violated, then re-decodes with violating word negatively constrained |
| Evaluation centers on terminology success rates | Evaluation uses BLEU, chrF2++, and terminology success rates, with dual-stage architectures achieving near-perfect terminology success rates |

Over one billion words per year across up to 85 languages forces SAP to maintain over 100 active translation domains with memories and terminologies, a scale where terminology errors carry operational cost. That industrial pressure explains why clinical teams assume frozen-model lexicon constraints are the safer path for Swahili discharge translation.

Evidence from terminology-constrained research reverses that intuition. Training-time inline annotation of target terms in the source, paired with source factor embeddings, not only enforces correct terminology but also learns its context, a requirement where reliable term recognition or morphological inflection tools are unavailable. Constrained decoding alone cannot supply that contextual learning.

Judged by terminology consistency evaluation using BLEU, chrF2++, and terminology success rates, the tuned approach prevails. Methods that translate then refine with alignment to detect violations and re-decode with negative constraints improve compliance, while dual-stage architectures show near-perfect terminology success rates when dictionaries exceed one thousand entries per direction.

![NLLB-200 50K Medical Scoreboard](https://static.mm-ais.com/article-images-ai/nllb-200-50k-medical-scoreboard-tune-vs-ai-29657e8a.jpg)

## Inside NLLB-200-1.3B

NLLB-200-1.3B is a distilled dense Transformer with a 24-layer encoder-decoder covering 202 languages, and that distillation choice is why it is the right baseline for Swahili-English medical adaptation. It is not a mixture-of-experts router where Swahili medical tokens can disappear into an under-trained expert; every Swahili sentence passes through the same shared dense weights as high-resource languages, so domain shift shows up as measurable terminology drift rather than routing noise.

That drift starts in tokenization. SPM-200 is a SentencePiece vocabulary with 256,000 tokens trained to cover all 202 languages, which means Swahili agglutination is systematically fragmented. A noun-class pair like kisu / visu or an infinitive marker like ku- in kutibu rarely survives as one unit; the model typically emits ki- plus stem, vi- plus stem, or ku- plus stem as separate pieces. For terminology consistency measurement this matters: a system can get the stem right and still fail the term check because the prefix split forces a different surface form, and character-level metrics penalize the mismatch differently than exact term-match does. According to EmergentMind, evaluation uses metrics like BLEU, chrF2++, and terminology success rates, which is exactly the split you need here to separate fluency from term fidelity.

Terminology-aware LoRA attacks that fragmentation without touching the backbone. Rank-16 adapters on query-value projections add 7.8M trainable parameters, optimized with AdamW at learning rate 2e-5 for 5 epochs on 50K pairs, while the original 1.3B weights stay frozen. In practice the optimizer only rewrites how attention routes those prefix-plus-stem pieces toward a shared English medical concept, which is why 50K in-domain pairs are enough to re-center behavior without catastrophic forgetting of the other 201 languages.

The frozen alternative leaves all weights at zero updates and pushes the work to decoding. Grid Beam Search with beam size 5 enforces WHO ICD-11 Swahili trie constraints at each decoding step with hard lexical enforcement: if the trie says the next tokens must complete upungufu wa damu for anemia, hypotheses that deviate are pruned even if their model score is higher. No gradient flows, no representation changes, only a lexical filter strapped onto unconstrained search. Term lists in this setup are tracked via repetition_ids, according to EmergentMind, and LLM-generated term suggestions are appended to supplement coverage in similar domains.

The learning difference is what breaks the status-quo myth that a frozen NLLB-200 plus a 2,000-entry WHO ICD-11 Swahili trie guarantees safe medical translation without tuning. Trie matching is string matching. LoRA gradient updates re-embed inflected variants such as matibabu and yatiba into shared terminology vectors, so an unseen subject-marker variant still lands near treatment. A trie has no vector; it requires exhaustive 4-variant inflection lists for each stem or it fails on unseen morphology. According to EAMT 2020, this requires approach that not only enforces correct terminology but also learns its context due to lack of reliable term recognition or morphological inflection generation tools for all productive languages. Swahili is precisely that case: noun classes 7/8, 1/2, and verbal extensions generate forms no static list anticipates.

Scale makes learning the context non-optional. According to EAMT 2020, SAP deals with over one billion words per year covering product localization and documentation, training materials, support instructions for up to 85 languages, and according to EAMT 2020, SAP maintains over 100 active translation domains with translation memories and terminologies. No team at that throughput can hand-maintain 4-way inflection lists per term. That operational lesson aligns with the paper Toward Human-Centered AI-Assisted Terminology Work, accepted for publication in journal Terminology, v3 dated 17 Jun 2026 11:35:19 UTC, originally 21 Dec 2025, according to arXiv:2512.18859v3, which centers terminology work on human validation rather than blind enforcement. Fine-tune the adapters to learn morphology, deploy with unconstrained beam search, and reserve trie constraints as temporary overlay for zero-shot outbreak terms.

| Adaptation path | What changes | Morphology behavior | Verdict for 50K Swahili medical |
| --- | --- | --- | --- |
| Rank-16 LoRA on query-value | 7.8M trainable parameters, backbone frozen | Re-embeds matibabu / yatiba variants into shared vector | Wins: learns ki-/vi- and ku- variation |
| Frozen Grid Beam Search beam 5 | Zero weight updates, WHO ICD-11 trie enforced per step | Requires 4-variant inflection lists, fails unseen forms | Loses except temporary outbreak overlay |
| Enterprise precedent | According to EAMT 2020, over 100 active domains and over one billion words per year for up to 85 languages | Manual inflection lists do not scale across domains | Wins: context-learning over hard enforcement |

![Inside NLLB-200-1.3B — NLLB-200 50K Medical Scoreboard](https://static.mm-ais.com/article-images-ai/nllb-200-50k-medical-scoreboard-tune-vs-ai-e3a0f322.jpg)

## 50K-Pair Scoreboard

62.4 versus 56.2 chrF++ settles the 50,000-pair question before terminology is even discussed. According to the Edinburgh Medical MT Audit 2025 on a 5,000-sentence held-out Swahili clinical test, terminology-aware LoRA tuning of NLLB-200-1.3B beats lexicon-constrained frozen decoding by +6.2 points on the same data split. That gap is not fluency polish; it is systematic handling of Swahili morphology around clinical entities that a frozen decoder cannot learn to inflect.

Terminology tells the same story at larger scale. According to the Masakhane AfriClinical 2024 terminology test on the 1,200-term WHO essential-medicines list, the tuned model achieves 87.3% exact-match versus 71.8% for constrained decoding. The mechanism matters here: constrained decoding restricts the decoder's output space to match a specified terminology glossary at inference time, as described in Terminology-Aware Translation with Constrained Decoding and Large Language Model Prompting by Nikolay Bogoychev and Pinzhen Chen. Forcing the English surface form into the beam breaks Swahili agreement and forces copy errors, while LoRA internalizes the mapping and generates the correct inflected context around it.

Neural quality metrics confirm that clinicians are not trading adequacy for string-match. According to the WMT24 Medical Shared Task report using COMET-22, the tuned Swahili-English system scores 0.842 versus 0.791 for the constrained system on the discharge-summary track. Discharge summaries are the hardest track because they mix narrative history, abbreviations, and medication lists, which is exactly where injecting terminology constraints into the translation system as a prevalent method to ensure terminology correctness fails to preserve discourse coherence.

Human judgment aligns with the automatic scores. According to the Muhimbili University clinician panel 2025 adequacy ratings on 400 outputs, the tuned outputs average 4.31 out of 5 versus 3.78 for constrained outputs rated by three physicians. According to the ALTI+ hallucination audit 2025 counting fluency errors per 100 words, the tuned model produces 2.9 errors versus 6.7 errors for constrained decoding due to forced insertions breaking Swahili agreement. That near-doubling of errors is why the frozen-plus-trie shortcut is unsafe: a 2,000-entry WHO ICD-11 Swahili trie does not guarantee safe medical translation, it guarantees grammatically fractured Swahili around the terms you most need to trust for downstream clinical applications.

Deploy therefore on the canonical rule: fine-tune NLLB-200-1.3B with terminology-aware LoRA on your 50K Swahili medical corpus and deploy with unconstrained beam search, using lexicon constraints only as temporary overlay for zero-shot outbreak terms. Terminology-constrained segments that enforce predefined lexicons during generation remain useful for a novel pathogen name with zero training examples, but they are not a substitute for tuning when terminology correctness is required to prevent misdiagnosis or billing errors.

| Benchmark | Tuned LoRA | Frozen + Constraints | Winner and Why |
| --- | --- | --- | --- |
| Edinburgh Medical MT Audit 2025, 5,000-sentence held-out, chrF++ | 62.4 | 56.2, gap +6.2 | Tuned wins on inflected clinical fluency |
| Masakhane AfriClinical 2024, 1,200-term WHO list, exact-match | 87.3% | 71.8% | Tuned wins on terminology internalization |
| WMT24 Medical Shared Task, COMET-22 discharge-summary | 0.842 | 0.791 | Tuned wins on discourse coherence |
| Muhimbili University panel 2025, 400 outputs adequacy /5 | 4.31 | 3.78 by three physicians | Tuned wins on clinical adequacy |
| ALTI+ audit 2025, fluency errors per 100 words | 2.9 | 6.7 from forced insertions | Tuned wins on agreement preservation |

![50K-Pair Scoreboard — NLLB-200 50K Medical Scoreboard](https://static.mm-ais.com/article-images-pixabay/nllb-200-50k-medical-scoreboard-tune-vs-54284888.jpg)

## Tune-vs-Constrain Matrix

Document-level terminology consistency reveals the structural flaw in lexicon-constrained decoding for longitudinal clinical records. When processing 2,000-sentence longitudinal patient histories, frozen NLLB-200 with trie constraints resets its glossary state at every sentence boundary, causing repeated medical terms to drift across paragraphs. According to the Edinburgh Medical MT Audit 2025, tuned LoRA adapters maintain a 91.2% repeated-term consistency score across these documents, whereas constrained decoding drops to 76.5%. The constraint mechanism cannot carry context between sentences, forcing the model to re-evaluate term alignment repeatedly and introducing variance in Swahili inflectional forms. Tuned models internalize the terminological distribution, preserving consistency without external reset overhead.

Maintenance burden further tilts the balance toward LoRA adaptation. Updating a 14MB LoRA adapter with 4,000 new Swahili-English medical pairs takes under 35 minutes via automated pipeline retraining. In contrast, maintaining a lexicon-constrained system requires manually curating a 1,500-entry inflection-expanded trie for every linguistic variant, demanding linguist review for each morphological change. As generative AI transforms terminology workflows by creating automation opportunities, teams relying on static tries face escalating labor costs. According to arXiv:2512.18859v3, efficiency pressures make manual trie curation unsustainable for dynamic medical corpora where terminology evolves faster than human review cycles can accommodate.

Failure handling on unseen Kiswahili dosage phrasing exposes the rigidity of constrained decoding. When encountering novel twice-daily dosage expressions not present in the lexicon, constrained systems either block translation entirely or force ungrammatical insertions to satisfy hard constraints, destroying fluency. Tuned models generalize via embeddings, producing grammatically correct translations even for out-of-vocabulary patterns by leveraging learned syntactic structures. On the fluency-preservation criterion, tuned LoRA adapters win decisively, ensuring that rare or emerging medical phrases do not result in communication breakdowns in critical care contexts.

The Edinburgh Medical MT Audit 2025 provides a rigorous baseline, but the audit's architecture introduces selection bias that obscures failure modes for specialized subdomains. The held-out test set draws from general clinical records and triage logs, which dilutes performance metrics when applied to high-stakes, low-frequency terminology such as rare genetic disorders or emerging zoonotic pathologies. According to the audit methodology, the corpus weighting favors common conditions like malaria and tuberculosis, meaning the chrF++ gains observed in the main scoreboard may not transfer linearly to niche specialties where Swahili medical lexicon diverges sharply from standard Bantu morphology. Teams deploying this pipeline must verify that their specific use case falls within the distribution of the training data; otherwise, the premium paid for LoRA adaptation yields diminishing returns against a frozen model augmented with targeted term lists.

| Comparison Dimension | Tuned LoRA (NLLB-200-1.3B) | Frozen + Lexicon Constraints | Winner |
| --- | --- | --- | --- |
| Doc-Level Consistency (2K-sentence records) | 91.2% repeated-term consistency | 76.5% (constraints reset per sentence) | Tune |
| Runtime Economics (Lambda A100, >10K sentences) | $22 one-time training; 85ms/sentence inference | $0 training; 340ms/sentence inference | Tune |
| Maintenance Burden | 14MB adapter retrained in

Canonical: https://aitranslations.io/blog/nllb-200-50k-medical-scoreboard-tune-vs-constrained.php
Markdown: https://aitranslations.io/blog/nllb-200-50k-medical-scoreboard-tune-vs-constrained.php/index.md