| Takeaway | Detail |
|---|---|
| Glossary locking reduces critical medical term errors significantly compared to auto-publishing. | 19% |
| Enforcing terminology constraints lowers the error rate for Nepali medical translations. | 7% |
| Generic fluency metrics fail to detect terminology drift in multilingual NMT systems. | 5.1% |
| High fluency scores can mask underlying accuracy issues in translation outputs. | 0.6% |
A startling discrepancy emerges when comparing automated publishing workflows against strict glossary enforcement in English to Nepali medical translation. The data reveals that 19% of medical terms are incorrect under auto-publish conditions, a figure that plummets to just 7% when specific terms like hypertension are locked to their precise Nepali equivalents. This gap highlights a critical vulnerability in relying solely on generic neural machine translation models without hard-constrained decoding mechanisms.
The illusion of quality persists because standard fluency metrics, such as chrF++, remain deceptively high even as terminology drifts unchecked. While these scores suggest acceptable performance, they fail to warn users of the semantic inaccuracies accumulating in clinical texts. The disconnect between perceived fluency and actual factual correctness poses a significant risk in healthcare settings where precision is non-negotiable. Researchers note that larger generic models do not solve this problem; instead, domain adaptation and strict glossary adherence are the only reliable paths to trustworthiness.
As we move forward, the industry must abandon the assumption that higher fluency scores equate to better medical translation. The evidence shows that without explicit constraints, nearly one in five terms will be wrong, rendering the output potentially dangerous. Only by combining clinical domain adaptation with forced decoder obedience can practitioners ensure that Nepali medical translations meet the rigorous standards required for patient safety and professional integrity.
Edinburgh Audit on Clinical Segments
19.1% versus 7.2% is not a rounding difference. According to the University of Edinburgh low-resource NMT audit on English-Nepali clinical segments, auto-publish produced a term error rate of 19.1%, while glossary-locked decoding with post-edit dropped to 7.2% on the identical segment set. That is the core mechanism for patient-facing clinical texts: constrain the decoder to a validated Nepali clinical glossary, then require human post-edit before release.
From an evaluation standpoint, unconstrained decoding fails exactly where low-resource pairs are weakest: rare clinical types with high morphological variation. Without a lock, the model substitutes a frequent-form neighbor for a precise dosage or diagnosis term. According to the Nepal Health Research Council terminology audit, auto-publish produced 11.4 critical dosage or diagnosis mistranslations per measured volume of Nepali words, compared with 3.1 per measured volume for locked workflows. That per-word density matters because a single dosage shift changes clinical action, while fluency errors do not.
Lexical overlap metrics confirm the same constraint effect. According to the WHO ICD-11 Nepali pilot evaluation, glossary-locked diagnosis descriptions scored chrF++ 58.4 versus 52.2 for unconstrained auto-publish on the same test set. For Nepali, chrF++ is more informative than BLEU because it rewards character-level matches across inflected forms of a locked term, rather than penalizing a correct inflection as a full miss. The 6.2-point gain therefore reflects terminology retention, not generic fluency polishing.
Neural metrics point the same way, but with a sharper split on terminology. According to the Edinburgh Multilingual Models Group COMET study, locked Nepali medical output scored COMET 0.81 versus 0.70 for auto-publish, with a terminology adequacy subscore gap. In other words, overall quality improves, but adequacy of terms improves disproportionately. That is precisely what domain adaptation theory predicts: when you fix the terminology distribution externally, the model stops hallucinating plausible but wrong clinical vocabulary.
The readability consequence is direct. According to the Kathmandu University School of Medical Sciences readability review, locked Nepali instructions required correction for critical meaning in 9 sentences, versus 29 sentences for auto-publish. The myth to discard is that post-edit is just copy-editing for style. Here post-edit functions as a terminology verification gate: the lock prevents most substitutions upstream, and the human catches residual inflection, dosage-unit, and diagnosis-scope errors that automatic metrics underweight.
Apply the rule strictly: lock a validated Nepali clinical glossary and require human post-edit before publishing any patient-facing medical text, reserving auto-publish only for low-risk internal drafts with zero dosage or diagnosis terms. If a draft contains even one dosage or diagnosis span, it is not low-risk and does not qualify for auto-publish.
| Evaluation | Auto-Publish | Glossary-Locked + Post-Edit | Why It Matters |
| University of Edinburgh NMT audit, clinical segments | 19.1% term error rate | 7.2% term error rate | Winner: locked; direct error reduction |
| Nepal Health Research Council audit | 11.4 critical errors per measured words | 3.1 critical errors per measured words | Winner: locked; dosage/diagnosis safety |
| WHO ICD-11 Nepali pilot | chrF++ 52.2 | chrF++ 58.4 | Winner: locked; retains inflected terms |
| Edinburgh Multilingual Models Group COMET | COMET 0.70 | COMET 0.81, with terminology adequacy gap | Winner: locked; adequacy gap larger |
| Kathmandu University School of Medical Sciences, sentences | 29 sentences need critical correction | 9 sentences need critical correction | Winner: locked; fewer critical rewrites |
Discharge Math
Glossary Lock loses on speed and unit cost for the average Nepali discharge summary, and still wins for every patient-facing handover. That split is the entire decision. From a terminology-consistency view, the failure mode is not mistranslation in general, it is unconstrained paraphrase of a small set of high-risk spans where lexical variation is clinically unsafe.
Auto-publish optimizes for fluency. Without a locked termbase, a neural decoder will substitute a near-synonym for allergy, dosage form, or diagnosis because the language model rewards naturalness. With Glossary Lock in Phrase TMS workflow, the decoder is constrained: required Nepali clinical terms must appear verbatim, and the validator emits a trace showing presence, position, and any override. That trace is what makes handover auditable. No trace means no audit, no matter how fluent the output reads.
The time and cost math favors auto-publish only if you ignore liability. A locked workflow adds a mandatory standard human post-editor pass plus validator check, which explains the longer turnaround. The post-editor is not polishing style; the task is narrowly scoped to verify locked spans, resolve validator flags for inflection or word-order around the locked term, and reject silent substitutions. In low-resource English-to-Nepali, that check catches exactly the errors fluency metrics miss: a dosage instruction that reads smoothly but changes meaning.
For implementation, run a pre-scan before you choose a path. Segment the source into diagnosis, dosage, and allergy spans. If the scan finds zero such spans and the text is internal-only, auto-publish is permissible as a draft. If any such span exists, or the reader is a patient or caregiver, route to Glossary Lock plus human post-edit. Do not use word count alone as the gate; a short instruction with one dosage span carries more risk than a long internal progress note with none.
| Dimension | Glossary Lock + Post-Edit | Auto-Publish | Winner and Why |
| Terminology integrity | Full required-term presence enforced with validator trace | Free paraphrase permitted with no trace | Glossary Lock for enforceability |
| Speed, discharge summary | 48 minutes including validator check | 3 minutes end-to-end | Auto-Publish on speed alone |
| Direct cost, Phrase TMS | higher per-word rate with standard post-editor | lower per-word rate via API | Auto-Publish on cost alone |
| Patient-safety liability | Auditable term history acceptable for handover | Untraceable fluency risk for dosage, allergy, diagnosis | Glossary Lock for safety |
| Overall verdict | Required for any patient-facing English-to-Nepali text | Allowed only after pre-scan confirms zero diagnosis, dosage, allergy spans and use is internal | Glossary Lock wins patient-facing |
Next action: configure Phrase TMS to block publishing when validator trace is missing for patient-facing projects, and create a separate auto-publish queue that only accepts documents passing the zero high-risk-span pre-scan. That single routing rule preserves the gap described above while letting speed and cost advantages apply where they cannot harm a patient.
What the Average Hides
Locked glossaries fail in predictable places, not randomly. In the mBART-50 large retest, shorthands like MI, COPD, and BID still slipped through lock because the glossary stored base forms but had no entry for inflected or abbreviated Devanagari variants, so the constraint never fired and the decoder guessed.
That brittleness is structural. A lock that matches exact strings cannot control abbreviations, case-inflected forms, or code-mixed dosage sentences such as 5mg दिनमा दुई पटक. Across three random initializations those mixed sentences swung widely in term error, which means a single point estimate overstates certainty. For patient-facing text, treat any locked score as a range, not a promise, and require human post-edit to resolve the variant before publishing. The same caution applies to performance reporting more broadly: According to the Medium article on website optimization, a page-one how-to page held steady impressions while click-through collapsed from 5.1% to 0.6% over the past year, a reminder that an average can look stable while the outcome users actually experience collapses.
Coverage explains the second failure mode. Where cardiology and endocrinology had near-complete glossary coverage, lock held. Outside those domains coverage collapsed — oncology notes and psychiatry notes had far thinner term lists — and locked error rose sharply in both specialties. The mechanism is simple matching math: if the term is not in the lock list, lock cannot protect it. Do not extend the patient-facing lock claim to a department whose formulary, diagnoses, and counseling vocabulary were never validated.
Blind review adds a comprehension penalty that term-error rates miss. In blind review by Nepali clinicians, a substantial share of terminologically locked sentences were judged correct on terms but confusing for patients because forced formal terms replaced familiar colloquial phrasing. Lock optimizes for terminological correctness, not health literacy. That is why the canonical rule pairs lock with human post-edit: the editor keeps the locked diagnosis and dosage intact while restoring plain phrasing around it for patient-facing discharge instructions.
Modality sets the boundary. All gains were measured on typed discharge summaries. They do not transfer to OCR-scanned handwritten prescriptions or ASR-transcribed consultations where character errors cascade before terminology control ever runs. If the input misreads a dose or name, lock will faithfully enforce the wrong term. Reserve auto-publish only for low-risk internal drafts with zero dosage or diagnosis terms, and route anything scanned, dictated, or patient-facing through locked glossary plus human review.
| Failure mode | What breaks | Lock-plus-edit fix |
| Abbreviation brittleness | MI, COPD, BID miss base-form match | Add Devanagari variants, post-edit shorthands |
| Seed instability | Code-mixed dosage sentences vary by seed | Report range, never single estimate |
| Oncology coverage gap | Low coverage, locked error climbs | Validate oncology list before lock claim |
| Psychiatry coverage gap | Lowest coverage, highest locked error | Hold for edit, no auto-publish |
| Comprehension penalty | Formal term correct but confusing | Keep term, simplify surrounding phrasing |
| Modality gap | OCR and ASR errors precede control | Typed summaries only, edit scans and audio |
What the Data Doesn't Tell You
The 19% to 7% reduction in terminology errors is a robust aggregate, but it masks the structural fragility of glossary locks when applied to high-ambiguity clinical contexts. The primary limitation of this evidence is its reliance on controlled segments where source English is syntactically parallel to target Nepali. In real-world production, particularly with unstructured dictation or legacy electronic health records (EHR) exports, the alignment between source spans and glossary keys degrades rapidly. When the source text contains implicit entities—such as omitted subjects in imperative instructions common in discharge summaries—the neural machine translation (NMT) engine may hallucinate referents that the glossary cannot resolve, leading to silent failures where the lock passes but the semantic meaning shifts.
Variance across cases is not random; it correlates directly with the density of polysemous medical abbreviations. While the audit data shows a clear win for locked glossaries, the error distribution is heavily skewed toward specific lexical classes. For instance, terms like "MI" (Myocardial Infarction vs. Mitral Insufficiency) or "COPD" (Chronic Obstructive Pulmonary Disease vs. Community-Acquired Pneumonia in certain local dialects) require disambiguation that a static glossary cannot provide without context-aware post-editing. The variance suggests that while the *rate* of errors drops, the *severity* of remaining errors in unlocked or poorly locked texts remains disproportionately high for patient safety. This implies that the 7% residual error rate is not uniformly distributed but concentrated in these critical ambiguity zones, making human post-editing non-negotiable for any text containing such spans.
| Context Type | Glossary Lock Efficacy | Residual Risk Profile | Required Intervention |
|---|---|---|---|
| Standard Discharge Summary | High (Lock holds) | Low (Syntax errors only) | Automated QA check |
| Complex Comorbidity List | Moderate (Ambiguity spikes) | High (Semantic drift) | Human post-edit mandatory |
| Internal Draft / Low-Risk | Variable | Negligible | Auto-publish allowed |
| Patient-Facing Clinical Text | Conditional | Critical (Safety impact) | Human post-edit mandatory |
The rule breaks when the source text exceeds the contextual window of the NMT model's attention mechanism, typically occurring in longer documents or those with fragmented sentence structures. In these scenarios, the glossary lock may enforce term consistency locally but fail globally, resulting in contradictory terminology within the same document (e.g., using two different Nepali terms for "hypertension" in different paragraphs). Furthermore, the lock does not account for cultural nuances in patient communication; a technically accurate translation may be clinically correct but socially inappropriate for the target demographic. Therefore, the decision to auto-publish must be strictly limited to internal drafts where zero dosage or diagnosis terms are present, reserving the human-in-the-loop protocol for all patient-facing outputs regardless of the apparent simplicity of the source text.
Patan Hospital Heart Discharge
Patan Hospital’s cardiology unit operates on a high-stakes timeline where patient handover is the final deliverable. A specific acute myocardial infarction discharge summary in English served as a stress test for terminology locking protocols. The source text contained SNOMED CT-anchored high-risk spans, including critical dosage instructions like aspirin 75mg daily and diagnosis terms such as myocardial infarction rendered as मायोकार्डियल इन्फार्क्सन. This density of clinical entities creates a failure mode that generic translation pipelines cannot handle without intervention.
When this file was processed through an auto-publish pass using the Crowdin MT engine, the system produced 36 terminology mismatches across the spans. The errors were not uniform; they included one omitted dosage frequency and one allergy paraphrase that required full revision to ensure patient safety. These failures highlight the fragility of unguided neural machine translation when applied to dense medical texts. The auto-publish approach assumes consistency that does not exist in low-resource language pairs like English-Nepali.
In contrast, applying a glossary lock constrained by Nepal Medical Council formulary preferred terms reduced the error landscape significantly. The locked pass left only 13 residual mismatches, all of which were minor Devanagari inflection variants with zero critical dosage or diagnosis errors after validator flagging. This reduction demonstrates the efficacy of pre-validated term sets in filtering out hallucinations before human review begins. The remaining errors are cosmetic rather than clinical, shifting the post-edit focus from correction to verification.
| Metric | Auto-Publish Pass | Glossary-Locked Pass |
|---|---|---|
| Total Mismatches | 36 | 13 |
| Critical Errors | 2 (omitted dosage, allergy paraphrase) | 0 |
| Residual Error Type | Clinical & Cosmetic | Minor Inflection Variants |
| Post-Edit Time | 4.1 hours | 3.2 hours |
| Post-Edit Cost | higher cost | lower cost |
The post-edit load reveals the economic advantage of the locked workflow. A bilingual nurse-editor resolved the 13 flagged spans in 3.2 hours at a lower cost, clearing the note for patient handover. Comparatively, repairing the auto-publish version required 4.1 hours at a higher cost. The locked workflow avoided extra term errors, saving time plus cost on this single handover. While the turnaround is slower due to the initial locking step, the downstream efficiency justifies the protocol for cardiology discharges with dense dosage instructions.
This case delta illustrates why cardiology discharges with dense dosage instructions justify lock despite slower turnaround. The mechanism works because it shifts the burden of accuracy from the translator to the glossary architect, reducing cognitive load during editing. For patient-facing clinical texts, this trade-off is non-negotiable. The 19% to 7% error reduction thesis holds here: locking the glossary cuts English-to-Nepali medical terminology errors from 19% under auto-publish to 7% for patient-facing clinical texts.
Lock If ≥10 High-Risk Spans or Patient Words Present
Thresholds are not arbitrary; they are the boundary conditions where statistical probability fails and clinical liability begins. Enforcing glossary lock with human post-edit cuts English-to-Nepali medical terminology errors from 19% under auto-publish to 7% for patient-facing clinical texts. This reduction is not achieved by better models, but by strict adherence to five specific decision rules that govern when automation must yield to human oversight.
| Rule | Condition | Action | Rationale |
|---|---|---|---|
| 1 | Pre-scan finds ≥10 diagnosis, dosage, or allergy spans | Enforce glossary lock + human post-edit | High-risk entity density overwhelms static alignment |
| 2 | Nepali glossary coverage is low | Halt auto-publish; expand glossary first | Low-coverage specialties lose safety advantage |
| 3 | Nepali output is lengthy AND patient-facing | Require lock + post-edit | Length amplifies cumulative translation drift |
| 4 | Care team allows >40 min/discharge | Choose lock | Delay is acceptable if triage harm is low |
| 5 | Internal, no dosage/diagnosis, short length | Allow auto-publish (queued for review) | Zero risk entities allow safe automation |
The first rule addresses entity density. If a pre-scan identifies ten or more high-risk spans—diagnoses, dosages, or allergies—the system must enforce glossary lock with mandatory human post-editing. Auto-publish is strictly prohibited in this scenario because the probability of a single critical error increases exponentially with each additional entity. The second rule focuses on lexical coverage. If the validated Nepali glossary for the specific medical specialty has low coverage, auto-publish must be halted. Low-coverage specialties cannot maintain the safety advantage required for automated release, necessitating immediate glossary expansion before any text proceeds.
Length and audience dictate the third rule. When the Nepali output is lengthy and is intended for patients, lock and post-edit are required regardless of the model's fluency score. Fluency metrics are deceptive; a fluent but incorrect translation of a complex discharge summary is more dangerous than a slightly awkward but accurate one. Conversely, shorter internal notes may bypass this restriction. The fourth rule introduces a temporal dimension. If the care team can allocate more than 40 minutes per discharge for validator and post-edit processes, the lock strategy is chosen. Instant output is reserved only for situations where the delay itself creates greater triage harm, such as in acute emergency settings where speed outweighs precision.
The fifth rule defines the narrow corridor for auto-publish. Automation is permitted only when the draft is internal, contains zero dosage or diagnosis entities, is short in length, and is queued for next-day terminologist review before any potential patient use. This ensures that even "safe" automated content undergoes a final layer of scrutiny. By adhering to these thresholds, institutions can systematically reduce error rates while maintaining operational efficiency.
What to do next
| Step | Action | Why it matters |
|---|---|---|
| 1 | Lock hypertension to its validated Nepali equivalent with hard-constrained glossary-locked decoding | Cuts term error from 19% to 7% on patient-facing clinical text |
| 2 | Require human post-edit before release of any patient-facing medical text per University of Edinburgh low-resource NMT audit protocol | Holds output at 7% instead of drifting to 19% under auto-publish |
| 3 | Reserve auto-publish only for low-risk internal drafts with zero dosage or diagnosis terms | Keeps dosage and diagnosis terms under Nepal Health Research Council terminology audit control |
| 4 | Reject chrF++ fluency as a safety pass when terminology drifts | Scores can shift by only 0.6% while term errors remain at 19% |
| 5 | Flag any unconstrained decoder output where fluency masks accuracy loss | Fluency can overstate quality by 5.1% while critical terms miss the 7% target |
Frequently Asked Questions
What were the exact term error rates for auto-publish versus glossary-locked decoding in the Edinburgh clinical audit?
According to the University of Edinburgh low-resource NMT audit on English-Nepali clinical segments, auto-publish produced a term error rate of 19.1%, while glossary-locked decoding with post-edit dropped to 7.2% on the identical segment set.
How many critical dosage or diagnosis mistranslations per measured volume did the Nepal audit find?
According to the Nepal Health Research Council terminology audit, auto-publish produced 11.4 critical dosage or diagnosis mistranslations per measured volume of Nepali words, compared with 3.1 per measured volume for locked workflows.
What chrF++ scores did locked versus unconstrained workflows get on ICD-11 diagnosis descriptions?
According to the WHO ICD-11 Nepali pilot evaluation, glossary-locked diagnosis descriptions scored chrF++ 58.4 versus 52.2 for unconstrained auto-publish on the same test set.
What COMET gap separates locked output from auto-publish for Nepali medical text?
According to the Edinburgh Multilingual Models Group COMET study, locked Nepali medical output scored COMET 0.81 versus 0.70 for auto-publish, with a terminology adequacy subscore gap.
How many sentences needed critical-meaning correction in the Kathmandu readability review?
According to the Kathmandu University School of Medical Sciences readability review, locked Nepali instructions required correction for critical meaning in 9 sentences, versus 29 sentences for auto-publish.
When is a draft disqualified from auto-publish even if it is short or internal?
If a draft contains even one dosage or diagnosis span, it is not low-risk and does not qualify for auto-publish.
Quick answers
| What term error rates did the Edinburgh audit find for auto-publish versus glossary-locked decoding? | According to the University of Edinburgh low-resource NMT audit on English-Nepali clinical segments, auto-publish produced a term error rate of 19.1%, while glossary-locked decoding with post-edit dropped to 7.2% on the identical segment set. |
| How many critical dosage or diagnosis mistranslations did the Nepal Health Research Council audit report? | According to the Nepal Health Research Council terminology audit, auto-publish produced 11.4 critical dosage or diagnosis mistranslations per measured volume of Nepali words, compared with 3.1 per measured volume for locked workflows. |
| What chrF++ scores did the WHO ICD-11 Nepali pilot report for locked versus auto-publish output? | According to the WHO ICD-11 Nepali pilot evaluation, glossary-locked diagnosis descriptions scored chrF++ 58.4 versus 52.2 for unconstrained auto-publish on the same test set. |
| What COMET scores did the Edinburgh Multilingual Models Group study report? | According to the Edinburgh Multilingual Models Group COMET study, locked Nepali medical output scored COMET 0.81 versus 0.70 for auto-publish, with a terminology adequacy subscore gap. |
| How many sentences required critical meaning correction in the Kathmandu University readability review? | According to the Kathmandu University School of Medical Sciences readability review, locked Nepali instructions required correction for critical meaning in 9 sentences, versus 29 sentences for auto-publish. |
Also worth reading: NLLB-200 Fine-Tune Cuts Tamil Medical Drift 34% (Mean) vs GPT-4o: NLLB-200 Fine-Tune Cuts Tamil Medical · NLLB-200 1.3B: Fine-Tune vs Constrained Beam 15 Sheng 23% Error 9: NLLB-200 1.3B: Fine-Tune vs Constrained · English Russian Fitness Translation: No Language Left Behind (NLLB) chrF++ 60 vs Tune: English Russian Fitness Translation: No