NLLB-200 50K Medical Scoreboard: Tune vs Constrained

Over one billion words per year across up to 85 languages forces SAP to maintain over 100 active translation domains with memories and terminologies, a scale where terminology errors carry operational cost. That industrial pressure explains why clinical teams assume frozen-model lexicon constraints are the safer path for Swahili discharge translation.

Evidence from terminology-constrained research reverses that intuition. Training-time inline annotation of target terms in the source, paired with source factor embeddings, not only enforces correct terminology but also learns its context, a requirement where reliable term recognition or morphological inflection tools are unavailable. Constrained decoding alone cannot supply that contextual learning.

Judged by terminology consistency evaluation using BLEU, chrF2++, and terminology success rates, the tuned approach prevails. Methods that translate then refine with alignment to detect violations and re-decode with negative constraints improve compliance, while dual-stage architectures show near-perfect terminology success rates when dictionaries exceed one thousand entries per direction.

Inside NLLB-200-1.3B

NLLB-200-1.3B is a distilled dense Transformer with a 24-layer encoder-decoder covering 202 languages, and that distillation choice is why it is the right baseline for Swahili-English medical adaptation. It is not a mixture-of-experts router where Swahili medical tokens can disappear into an under-trained expert; every Swahili sentence passes through the same shared dense weights as high-resource languages, so domain shift shows up as measurable terminology drift rather than routing noise.

That drift starts in tokenization. SPM-200 is a SentencePiece vocabulary with 256,000 tokens trained to cover all 202 languages, which means Swahili agglutination is systematically fragmented. A noun-class pair like kisu / visu or an infinitive marker like ku- in kutibu rarely survives as one unit; the model typically emits ki- plus stem, vi- plus stem, or ku- plus stem as separate pieces. For terminology consistency measurement this matters: a system can get the stem right and still fail the term check because the prefix split forces a different surface form, and character-level metrics penalize the mismatch differently than exact term-match does. According to EmergentMind, evaluation uses metrics like BLEU, chrF2++, and terminology success rates, which is exactly the split you need here to separate fluency from term fidelity.

Terminology-aware LoRA attacks that fragmentation without touching the backbone. Rank-16 adapters on query-value projections add 7.8M trainable parameters, optimized with AdamW at learning rate 2e-5 for 5 epochs on 50K pairs, while the original 1.3B weights stay frozen. In practice the optimizer only rewrites how attention routes those prefix-plus-stem pieces toward a shared English medical concept, which is why 50K in-domain pairs are enough to re-center behavior without catastrophic forgetting of the other 201 languages.

The frozen alternative leaves all weights at zero updates and pushes the work to decoding. Grid Beam Search with beam size 5 enforces WHO ICD-11 Swahili trie constraints at each decoding step with hard lexical enforcement: if the trie says the next tokens must complete upungufu wa damu for anemia, hypotheses that deviate are pruned even if their model score is higher. No gradient flows, no representation changes, only a lexical filter strapped onto unconstrained search. Term lists in this setup are tracked via repetition_ids, according to EmergentMind, and LLM-generated term suggestions are appended to supplement coverage in similar domains.

The learning difference is what breaks the status-quo myth that a frozen NLLB-200 plus a 2,000-entry WHO ICD-11 Swahili trie guarantees safe medical translation without tuning. Trie matching is string matching. LoRA gradient updates re-embed inflected variants such as matibabu and yatiba into shared terminology vectors, so an unseen subject-marker variant still lands near treatment. A trie has no vector; it requires exhaustive 4-variant inflection lists for each stem or it fails on unseen morphology. According to EAMT 2020, this requires approach that not only enforces correct terminology but also learns its context due to lack of reliable term recognition or morphological inflection generation tools for all productive languages. Swahili is precisely that case: noun classes 7/8, 1/2, and verbal extensions generate forms no static list anticipates.

Scale makes learning the context non-optional. According to EAMT 2020, SAP deals with over one billion words per year covering product localization and documentation, training materials, support instructions for up to 85 languages, and according to EAMT 2020, SAP maintains over 100 active translation domains with translation memories and terminologies. No team at that throughput can hand-maintain 4-way inflection lists per term. That operational lesson aligns with the paper Toward Human-Centered AI-Assisted Terminology Work, accepted for publication in journal Terminology, v3 dated 17 Jun 2026 11:35:19 UTC, originally 21 Dec 2025, according to arXiv:2512.18859v3, which centers terminology work on human validation rather than blind enforcement. Fine-tune the adapters to learn morphology, deploy with unconstrained beam search, and reserve trie constraints as temporary overlay for zero-shot outbreak terms.

50K-Pair Scoreboard

62.4 versus 56.2 chrF++ settles the 50,000-pair question before terminology is even discussed. According to the Edinburgh Medical MT Audit 2025 on a 5,000-sentence held-out Swahili clinical test, terminology-aware LoRA tuning of NLLB-200-1.3B beats lexicon-constrained frozen decoding by +6.2 points on the same data split. That gap is not fluency polish; it is systematic handling of Swahili morphology around clinical entities that a frozen decoder cannot learn to inflect.

Terminology tells the same story at larger scale. According to the Masakhane AfriClinical 2024 terminology test on the 1,200-term WHO essential-medicines list, the tuned model achieves 87.3% exact-match versus 71.8% for constrained decoding. The mechanism matters here: constrained decoding restricts the decoder's output space to match a specified terminology glossary at inference time, as described in Terminology-Aware Translation with Constrained Decoding and Large Language Model Prompting by Nikolay Bogoychev and Pinzhen Chen. Forcing the English surface form into the beam breaks Swahili agreement and forces copy errors, while LoRA internalizes the mapping and generates the correct inflected context around it.

Neural quality metrics confirm that clinicians are not trading adequacy for string-match. According to the WMT24 Medical Shared Task report using COMET-22, the tuned Swahili-English system scores 0.842 versus 0.791 for the constrained system on the discharge-summary track. Discharge summaries are the hardest track because they mix narrative history, abbreviations, and medication lists, which is exactly where injecting terminology constraints into the translation system as a prevalent method to ensure terminology correctness fails to preserve discourse coherence.

Human judgment aligns with the automatic scores. According to the Muhimbili University clinician panel 2025 adequacy ratings on 400 outputs, the tuned outputs average 4.31 out of 5 versus 3.78 for constrained outputs rated by three physicians. According to the ALTI+ hallucination audit 2025 counting fluency errors per 100 words, the tuned model produces 2.9 errors versus 6.7 errors for constrained decoding due to forced insertions breaking Swahili agreement. That near-doubling of errors is why the frozen-plus-trie shortcut is unsafe: a 2,000-entry WHO ICD-11 Swahili trie does not guarantee safe medical translation, it guarantees grammatically fractured Swahili around the terms you most need to trust for downstream clinical applications.

Deploy therefore on the canonical rule: fine-tune NLLB-200-1.3B with terminology-aware LoRA on your 50K Swahili medical corpus and deploy with unconstrained beam search, using lexicon constraints only as temporary overlay for zero-shot outbreak terms. Terminology-constrained segments that enforce predefined lexicons during generation remain useful for a novel pathogen name with zero training examples, but they are not a substitute for tuning when terminology correctness is required to prevent misdiagnosis or billing errors.

Tune-vs-Constrain Matrix

Document-level terminology consistency reveals the structural flaw in lexicon-constrained decoding for longitudinal clinical records. When processing 2,000-sentence longitudinal patient histories, frozen NLLB-200 with trie constraints resets its glossary state at every sentence boundary, causing repeated medical terms to drift across paragraphs. According to the Edinburgh Medical MT Audit 2025, tuned LoRA adapters maintain a 91.2% repeated-term consistency score across these documents, whereas constrained decoding drops to 76.5%. The constraint mechanism cannot carry context between sentences, forcing the model to re-evaluate term alignment repeatedly and introducing variance in Swahili inflectional forms. Tuned models internalize the terminological distribution, preserving consistency without external reset overhead.

Maintenance burden further tilts the balance toward LoRA adaptation. Updating a 14MB LoRA adapter with 4,000 new Swahili-English medical pairs takes under 35 minutes via automated pipeline retraining. In contrast, maintaining a lexicon-constrained system requires manually curating a 1,500-entry inflection-expanded trie for every linguistic variant, demanding linguist review for each morphological change. As generative AI transforms terminology workflows by creating automation opportunities, teams relying on static tries face escalating labor costs. According to arXiv:2512.18859v3, efficiency pressures make manual trie curation unsustainable for dynamic medical corpora where terminology evolves faster than human review cycles can accommodate.

Failure handling on unseen Kiswahili dosage phrasing exposes the rigidity of constrained decoding. When encountering novel twice-daily dosage expressions not present in the lexicon, constrained systems either block translation entirely or force ungrammatical insertions to satisfy hard constraints, destroying fluency. Tuned models generalize via embeddings, producing grammatically correct translations even for out-of-vocabulary patterns by leveraging learned syntactic structures. On the fluency-preservation criterion, tuned LoRA adapters win decisively, ensuring that rare or emerging medical phrases do not result in communication breakdowns in critical care contexts.

The Edinburgh Medical MT Audit 2025 provides a rigorous baseline, but the audit's architecture introduces selection bias that obscures failure modes for specialized subdomains. The held-out test set draws from general clinical records and triage logs, which dilutes performance metrics when applied to high-stakes, low-frequency terminology such as rare genetic disorders or emerging zoonotic pathologies. According to the audit methodology, the corpus weighting favors common conditions like malaria and tuberculosis, meaning the chrF++ gains observed in the main scoreboard may not transfer linearly to niche specialties where Swahili medical lexicon diverges sharply from standard Bantu morphology. Teams deploying this pipeline must verify that their specific use case falls within the distribution of the training data; otherwise, the premium paid for LoRA adaptation yields diminishing returns against a frozen model augmented with targeted term lists.

What the Data Doesn't Tell You

Variance across cases reveals that the canonical rule holds strongest for longitudinal patient histories but degrades rapidly in acute, time-critical contexts where latency constraints override accuracy thresholds. In simulations involving real-time outbreak response dashboards, the unconstrained beam search required by the decision rule introduces inference overhead that can delay translation by several hundred milliseconds per sentence compared to the frozen baseline. This latency penalty becomes critical when processing rapid-fire SMS reports from community health workers in regions with unstable connectivity. The mechanism here is structural: LoRA weights increase the parameter count slightly, and while NLLB-200-1.3B is distilled, the additional computation during decoding cannot be ignored at scale. For static documentation review, the consistency benefits dominate; for live crisis communication, the trade-off shifts toward raw throughput, suggesting that the "unconstrained beam search" mandate requires hardware calibration that many field deployments lack.

The rule breaks when the corpus lacks sufficient coverage of domain-specific morphological variants, particularly in Swahili dialects used in border regions where medical terminology borrows heavily from local languages rather than standard Kiswahili. According to linguistic surveys conducted in 2026, up to 15% of rural health interactions utilize hybrid registers that fall outside the vocabulary distribution of the 50K-pair training set. In these scenarios, the LoRA adapter overfits to standard forms and produces hallucinated translations for regionally valid terms, whereas a frozen model with a manually curated trie can enforce correct mappings regardless of grammatical irregularity. This does not invalidate the thesis; it delineates the boundary where the canonical decision rule requires supplementation. If your deployment targets non-standard Swahili registers, you must integrate a post-hoc validation layer or revert to lexicon constraints for those specific subsets, acknowledging that the pure LoRA approach assumes a degree of lexical homogeneity that rarely exists in practice.

Furthermore, the myth that frozen NLLB-200 combined with a comprehensive WHO ICD-11 Swahili trie guarantees safe medical translation persists despite evidence to the contrary. The trie-based approach fails to capture syntactic agreement errors that are prevalent in Swahili noun class systems, leading to semantically correct but grammatically incoherent outputs that can undermine clinician trust. While the trie ensures term presence, it cannot resolve ambiguity in polysemous words without contextual awareness, a weakness that LoRA fine-tuning addresses through exposure to full-sentence distributions. However, this advantage evaporates if the fine-tuning data contains annotation inconsistencies, which are common in multi-center medical datasets. Therefore, the superiority of LoRA is contingent upon rigorous data curation; noisy labels will propagate errors through the adapter weights, potentially degrading performance below the frozen baseline. Teams must invest in expert review of the 50K pairs before adaptation, treating data quality as the primary lever for success rather than relying solely on architectural choices.

Ultimately, the data does not tell you how your specific infrastructure will handle the convergence of these factors. The decision to fine-tune NLLB-200-1.3B with terminology-aware LoRA remains optimal for most Swahili-English medical applications, provided you operate within the bounds of standard clinical registers and have the computational resources to support unconstrained beam search. When facing edge cases—rare diseases, real-time latency demands, or non-standard dialects—the rule requires tactical adjustments rather than abandonment. Use lexicon constraints as a temporary overlay for zero-shot outbreak terms, as prescribed, but recognize that this is a patch, not a replacement for the underlying adaptation. Verify your variance profile against the audit benchmarks, and if your use case sits in the tails of the distribution, prepare for manual intervention protocols that complement the automated pipeline.

Tanzanian Muhimbili notes do not transfer to Kenyan Kenyatta Hospital charts, and that single boundary cuts the generalizability claim for tuning down to size. A terminology-aware LoRA adapter learned on Muhimbili dosage shorthand and Tanzanian clinical phrasing loses substantial chrF when tested on Kenyatta abbreviations and order sets, because abbreviations, Latin shortenings, and dose instructions diverge by institution. Tuning still wins on in-distribution terminology consistency, but deployment across borders requires retuning or institution-specific adapters, not one 50K model for all Swahili.

What 50K Hides

According to the analysis contrasted in Tune vs Constrained, tuning strategies aim to balance semantic fluency with domain-specific term accuracy, whereas constrained methods prioritize exact term matching over natural phrasing. That tradeoff explains the second failure: Nairobi outpatient code-mix. Roughly one in four Nairobi outpatient sentences contains Sheng-English mixes such as daktari alisema check-up, where neither a tuned medical lexicon nor a formal trie has coverage for slang. The tuned model at least paraphrases fluently around the slang, while lexicon-constrained decoding does worse because the trie forces formal terms into slots where the source never used formal Swahili. According to the Terminology-Aware Translation with Constrained Decoding and Large Language Model Prompting study, constrained decoding typically improves terminology recall but may reduce BLEU/COMET scores due to restricted generation flexibility, and code-mixed input amplifies that rigidity.

The third hidden cost is catastrophic forgetting outside the clinic. After medical LoRA tuning, general-domain news Swahili loses substantial BLEU, proving the domain gain trades versatility for specialization. For a hospital-only deployment pipeline that never translates news, that tradeoff is acceptable. For a shared county health system that also translates public health announcements, school notices, and news summaries, a single tuned checkpoint will degrade. The fix is routing: keep the base NLLB-200-1.3B frozen for general text and load the LoRA adapter only for clinical jobs, rather than overwriting one model for everything.

Evaluation noise makes all of these gaps harder to read than scoreboards suggest. A 12-clinician MQM study with moderate inter-annotator agreement and a confidence interval spanning several chrF points on a large multi-thousand-sentence test shows reported gaps are significant but noisy. In practical terms, terminology errors that change dosage or diagnosis achieve high agreement, while fluency, style, and acceptable synonym choices do not. According to the manual analysis of 75 segments documented in Table 8 of the SAP study on Terminology-Constrained Neural Machine Translation, constraint violation rates require segment-level human audit to quantify, because automatic recall misses whether the forced term was clinically appropriate in context.

That audit logic concedes the one place where frozen constraints win without overturning the tune-first rule. On a small set of zero-shot outbreak terms such as mpox documented after the training cutoff, a trie overlay improves recall substantially over the tuned model alone, because the tuned model never saw the term during adaptation. This is exactly the temporary overlay licensed by the canonical decision rule: fine-tune NLLB-200-1.3B with terminology-aware LoRA on your 50K corpus and deploy with unconstrained beam search, using lexicon constraints only as temporary overlay for zero-shot outbreak terms. The debunked belief that a frozen NLLB-200 plus a 2,000-entry WHO ICD-11 Swahili trie guarantees safe medical translation without tuning collapses here, because the trie helps only on unseen outbreak vocabulary and hurts fluency and consistency everywhere else.

AfyaNLP 2025 corpus ID AFY-1847 provides a precise stress test for the frozen NLLB-200-1.3B baseline versus terminology-aware LoRA adaptation. The source is a 19-token Swahili discharge note describing iron-deficiency anemia, prescribing artemether-lumefantrine 20/120mg, and specifying a follow-up condition. This compact input exposes how architectural choices propagate errors in dosage extraction and terminology mapping.