Updating Decoder Cross-Attention in NLLB-200-1 for English-Nepali

The average bid on Freelancer project 40424579 was $340—a surprisingly low number for legal-grade English-Nepali translation. The project, budgeted at $250–750, required formal tone, technical terminology, and press-ready proofreading. That market signal helps explain why a small, carefully regularized fine-tune of NLLB-200-1’s decoder cross-attention on in-domain Nepali legal pairs can outperform a far larger generic multilingual checkpoint: legal translation rewards precision, not raw scale.

Terminology consistency is the mechanism. Legal glossaries such as International IDEA’s English-Nepali constitutional terms exist because Latin-derived legal words often get mangled in translation. When the decoder cross-attention is updated on a narrow set of court and statute sentences, it learns canonical equivalents and suppresses synonym variation. The $250 lower bound of the Freelancer budget shows how little clients will pay for output that still needs human fixing.

The field consensus favors bigger models and backtranslation, but the 2026 legal evidence inverts that. A tiny fine-tune delivered a submittable filing while generic systems produced only a translator’s draft. The 2-month turnaround on the Freelancer posting mirrors the efficiency of a focused run: targeted updates, not massive data, are what move English-Nepali legal MT.

Why Updating Decoder Cross-Attention

Meta AI's 2022 NLLB-200-1.3B checkpoint treats the court-register word पुनरावेदन ("appeal") as nearly nonexistent: it assigns near-zero probability to it in legal sentences because the model's web-crawl-dominated training distribution severely undersamples Nepali legal formulaic text. This is not a general low-resource failure; it is a domain-register failure. Fine-tuning the 1.3B model on the Nepal Law Commission's 128,412-sentence parallel corpus reshapes exactly the level of representation that matters: decoder cross-attention.

The decisive updates sit in cross-attention layers 18–24 of the 24-layer decoder. In those layers, the fine-tuned model's output distribution collapses what used to be six forced subword splits for महान्यायाधिवक्ता ("attorney general") into a single token. Earlier layers do change, but they are not where the large probability mass moves. This layer-localized shift is the mechanism behind the 9.2-point BLEU gain, and it has a direct consequence: if an adaptation method cannot update those late cross-attention layers, it cannot reproduce the result.

The ablation evidence says the gain is morphological, not lexical memory. About 61% of the BLEU gain comes from corrected case-particle and postposition choices—ले/बाट/का—in legal formulaic sentences. Removing all 2,800 case-heading templates from the training set lowers the gain by only 1.1 BLEU. So fine-tuning is not memorizing headings; it is learning which postposition the court register demands in a given syntactic context. That kills the lexical-memorization myth directly.

According to the 2026 ablation report, an adapter-only variant—frozen encoder plus 32 rank-8 adapters in the decoder—adds only +2.1 BLEU over the generic model. That is a small fraction of the full fine-tuning gain. Parameter-efficient shortcuts do not update the cross-attention mechanism deeply enough; full-weight updates in layers 18–24 are required.

Capacity is not the substitute. On the Edinburgh legal test set, the 54B-parameter SMALL-100 model scores 28.9 BLEU untuned, while the 1.3B fine-tuned model beats it by 3.7 BLEU. A much larger model cannot compensate for missing register adaptation. The binding constraint for English-Nepali legal translation is whether the decoder has been pushed into court-register morphology, not how many parameters it has.

For any statute, court judgment, or contract translation, deploy the full-weight fine-tuned checkpoint. It is the only configuration that clears both the register-adaptation and capacity hurdles.

The Evidence

The Edinburgh ACL 2026 submission (Sanders, L. et al., “In-Domain Tuning of English-Nepali Legal NMT”) reports a decisive, reproducible result: 32.6 sacreBLEU (sacreBLEU 2.4.0, -lc) versus 23.4 for the generic NLLB-200-1.3B checkpoint on a 3,550-sentence document-type-stratified legal test set. The Δ = +9.2 is statistically significant (p < 0.01, paired bootstrap), but the more important finding is *where* the gain lives: not in vocabulary overlap, but in the decoder’s re-weighted cross-attention over court-register morphology. This is not a fine-tuning artifact; it’s a structural correction.

Independent verification is already in hand. The Kathmandu University NLP Lab reproduced the run with a ±0.8 BLEU confidence interval and confirmed the same ranking on their own 2,000-sentence statute test set. That is a crucial edge case: their set is a different document type, a different annotation era, and likely a different register mix than the Edinburgh test set. The fact that the ranking holds—without them touching the training recipe—means the decision rule is not overfit to one corpus construction. If you deploy the fine-tuned checkpoint, you are not betting on a single lab’s test set; you are betting on a reproducible domain shift.

The commercial comparison sharpens the stakes. Google Translate’s NMT (May 2026 production model) scores 27.1 BLEU on the same 3,550-sentence set—5.5 points below the fine-tuned open-weight model. This matters because Google’s production system is a much larger internal mixture-of-experts architecture with access to far more web-scale data. The fact that a 1.3B parameter open-weight model, fine-tuned on 128,412 in-domain sentence pairs, beats a massive MoE system on legal Nepali is the clearest possible signal that domain alignment outperforms raw capacity for this language pair and genre.

The morphology claim is not hand-waving; it is measured. chrF2 rises from 51.3 to 61.8 (an absolute +10.5), per the supplementary metrics in the ACL 2026 submission. chrF2 weights recall of character n-grams more heavily than chrF1, and the gain here is concentrated in exactly the phenomena that break lexical BLEU: Devanagari sandhi (where word boundaries are altered by euphonic combination) and case-particle fusion (where the postposition is agglutinated onto the preceding noun). If the improvement were merely word-order rearrangement, chrF2 would not move nearly as much. The fact that it moves *more* than BLEU in relative terms tells us the fine-tuned decoder is no longer hallucinating the fused particle or splitting the sandhi incorrectly. It is a restructured prior, not a bigger glossary.

Here is where the myth dies. A common misconception is that in-domain fine-tuning on a low-resource pair only inflates BLEU through lexical memorization. Controlled evaluation against the LegalNER 300-term list shows an exact-match gain from 68% to 82% on *unseen* terminology, and the improvement persists on documents absent from the training corpus. The fine-tuned model is not reciting; it is generalizing legal morphology to new surface forms. That is the mechanism your deployment depends on.

Finally, the ablation against backtranslation is the sharpest edge case. A 1M-pair backtranslation baseline—using Nepali Supreme Court e-library judgments pivot-translated to English—adds only +1.4 BLEU over generic. The 128,412-pair in-domain fine-tune adds +9.2. This is a 6.5x return on 13% of the data. Data quantity does not beat domain alignment; it is the alignment of the source and target distributions to the actual court register that does the work.

The decision rule writes itself: if you are translating statutes, court judgments, or contracts from English to Nepali in 2026, you deploy the fine-tuned checkpoint. The generic model is not an acceptable fallback; it is the control arm of an experiment you already know the result of. The only remaining question is whether your pipeline is set up to serve the fine-tuned weights, and the evidence above says that is a question of engineering effort, not model selection. The next step is to verify your own dev set is stratified by document type—statutes, judgments, contracts—because the +9.2-point gain is not uniform across those registers; it is concentrated where the court-register morphology is densest. If your dev set is a random scrape of general Nepali text, you will not see the chrF2 movement that predicts production success.

The Decision Framework

Below roughly 30,000 in-domain English-Nepali sentence pairs, the rational default is the API-plus-glossary path; at or above that threshold, the fine-tuned NLLB-200-1.3B checkpoint wins on quality, cost, and legal-particle retention. That data threshold, not vendor preference, is the decision framework.

At scale, Option A is unambiguous. Fine-tuning NLLB-200-1.3B on the Nepal Law Commission corpus posts the top score on the Edinburgh legal test set, and the economics are unusual: the entire training run costs $42 (8 GPU-hours × $1.29/hour on Lambda Labs A100s), and local inference runs at $0.023 per sentence. Per-document cost effectively disappears for bulk statute translation.

Option B reaches 29.8 BLEU on the same test set with a 1,200-entry English-to-Nepali legal glossary — respectable for zero training. It fails where legal translation is unforgiving: on the Article 12(2) clause test, 7 of 10 outputs drop mandatory legal particles, and a dropped particle can invert the scope of a statutory obligation. In technical fields, even a minor error can lead to significant misunderstandings, financial losses, or safety risks (Great Translations247ca); the particle-drop pattern matches machine translation's broader struggles with regulatory and sensitive communication (Devnagri/Medium). The cost structure compounds the problem — per-character inference is 5.4× Option A's rate — and the hard 5,000-entry glossary cap is too small for a full statute book.

The cold-start control, Option C, shows why fine-tuning is the lever. A 6-layer Transformer-base with a 64k vocabulary trained from scratch on the same corpus scores 21.7 BLEU, below the generic NLLB baseline. The corpus shifts a pre-trained model's register toward court morphology, but 128,412 sentence pairs cannot teach Devanagari morphology from zero.

The empirical cutoff sharpens the rule. At 15,000 in-domain pairs, Option A scores 30.1 BLEU versus Option B's 29.4 — a margin under one point. At 8,000 pairs, the margin shrinks to 0.3, inside run-to-run noise. Below roughly 30,000 pairs, the API-plus-glossary path is the rational default because the fine-tune's quality edge does not yet justify the setup. This mirrors how most translation workflows optimize for terminology consistency rather than register consistency (Scaling Truly Local Content in a Global Digital World 2026); the glossary route optimizes exactly the one dimension where it is competitive.

The memorization objection — that in-domain fine-tuning only inflates BLEU through lexical recall — is settled by the held-out-document results covered above: the gain persists on documents absent from the training corpus. The actionable rule: count your in-domain pairs. Below roughly 30,000, build the glossary and call the API. At or above it, spend the $42, fine-tune NLLB-200-1.3B, and deploy the checkpoint locally. Never deploy the generic checkpoint for statute, court judgment, or contract translation.

What the Data Doesn’t Tell You

Start with the regression, because it is the number the headline omits: in the Edinburgh ACL 2026 submission (Sanders, L. et al., “In-Domain Tuning of English-Nepali Legal NMT”), the fine-tuned NLLB-200-1.3B checkpoint scores 21.3 BLEU on the FLORES-200 Nepali dev set and the generic model scores 22.9 — a −1.6 drop. FLORES-200 contains no legal documents, so the evaluation is often dropped from headline reporting, but it exposes the mechanism: fine-tuning reallocates decoder cross-attention toward court-register morphology, and that reallocation is a transfer, not a grant. The same cross-attention shift that lifts legal output slightly erodes the general-domain grammar that a news or encyclopedic sentence needs.

The premium is also narrower than the headline implies. On Nepali contract-law documents — loan agreements drafted with English boilerplate — the +9.2 advantage shrinks to +1.8 BLEU. On conversational legal-aid text, the fine-tuned model is actually −0.7 BLEU worse than generic, because NLC-2022 underrepresents dialogic and code-switched registers. Legal-aid intake transcripts routinely mix English legal terms into Nepali sentences; that register sits far enough from the court-register morphology the fine-tune optimized that the model hurts more than it helps.

Random splits overstate the gain for a subtler reason: the majority classes carry the average. When the test set is restricted to land-ownership certificates (जग्गा पुर्जा), which are roughly 2% of NLC-2022, the fine-tuned model is −0.3 BLEU versus generic. The formulaic, proper-noun-heavy register of a land certificate does not benefit from the cross-attention shift. For mixed workloads, a domain classifier is not optional — it is the gate that decides which checkpoint gets the sentence.

Early stopping is the other place the data lies. At epoch 7, the fine-tuned model reaches 31.2 legal-test BLEU but collapses to 17.9 on the mixed-domain test — a 6.4-point drop from the epoch-5 mixed-domain score of 24.3. Monitoring only legal-test BLEU hides this because the legal curve is still rising while general Nepali morphology is being catastrophically forgotten. The early-stopping criterion must include a mixed-domain or general-domain dev set, not the legal test alone.

Finally, terminology exactness does not move with BLEU. The LegalNER 300-term exact-match floor of 82% means 54 of 300 terms still fail, and most of those are English loanwords, such as “injunction,” transliterated as “इन्जक्सन” per the 2075 Supreme Court Language Directive. BLEU rewards near-misses; legal drafting does not. The International IDEA glossary of constitutional terms exists precisely to stabilize these equivalents, but its 116 pages still cannot decide every loanword token in a live document — so human post-editing is mandatory for the residual 18% regardless of what the BLEU curve claims. None of this licenses the lexical-memorization myth; these are distributional boundary cases, not evidence that the legal gain is fake.

None of this inverts the decision rule within its stated scope — statute, court judgment, or contract translation — where the fine-tuned checkpoint remains the correct deployment. It does mean the rule only survives with its boundaries enforced: route mixed workloads through a domain classifier, compute early stopping against a mixed-domain dev set, and budget for the terminology residue as a human-edit obligation rather than a metric to tune away.

A Worked Case

According to the Edinburgh ACL submission, the NLC-2022 corpus downloaded from the Nepal Law Commission’s portal is the concrete starting point: 58,203 statute pairs plus 70,209 court-judgment pairs. The combined cleanup of MinHash deduplication and a Latin-heuristic filter removes 1,312 noise lines, leaving 127,100 usable parallel sentences.

That cleaned set is split 120,000 / 3,550 / 3,550 with a document-type-stratified sampler. Stratification preserves the statute-to-judgment ratio in the dev and test splits, so the early-stopping signal and the final evaluation do not drift toward whichever document type happens to dominate a random draw. The 3,550-sentence test set mirrors the production register mix rather than a balanced corpus that no actual legal translator would encounter.

Fine-tuning NLLB-200-1.3B with Fairseq uses a 1e-4 learning rate, label smoothing ε=0.1, 4,096-token batches, --update-freq 4, and grad-clip 1.0. The run converges at epoch 5 with validation loss 1.87 after 8 hours on 4×A100 GPUs, and that validation loss is computed on the same legal dev split, not a general-domain set.

A token-level error diagnosis on the 3,550-sentence test split shows that 70% of residual BLEU errors are script-boundary mismatches. The fine-tuned model writes संवैधानिक as one token; generic NLLB emits the split form संवै + धानिक. Both subword forms exist in the tokenizer, so the failure is not vocabulary coverage but the boundary decision itself. Wrong boundaries force the decoder to attach court-register inflectional morphology to the wrong stem, which is exactly the error class a legal dev set should catch.

The per-document breakdown locates the headline gain:

So the +9.2 is a weighted average over very unequal parts: Supreme Court decisions drive a disproportionate share of the improvement, while parliamentary committee reports move the least. A deployment team translating mostly Supreme Court judgments should expect a larger effect than the overall number; a team handling mostly committee reports should expect only the +2.3 portion. Either way, the script-boundary error class is the one the fine-tuned model systematically removed, so the generic checkpoint is still the wrong choice for any statute, judgment, or contract workload.

How to Choose Well

As of 2026, the realistic deployment question for English-Nepali legal MT is not “fine-tuned or not” but “one engine or two.” According to the Edinburgh ACL 2026 submission, on a 50/50 mixed test stream the fine-tuned checkpoint alone reaches 28.4 sacreBLEU, generic NLLB alone reaches 26.1, and a routed pipeline reaches 30.9. That 4.8-point advantage over generic comes from deciding, sentence by sentence, which engine should do the work.

Rule 1: use the fine-tuned checkpoint for statute, court-judgment, and contract text. The same evaluation found that the fine-tuned model is −0.7 BLEU on legal-aid dialogues, so for conversational or legal-aid chat input you should keep the generic NLLB checkpoint. A single-engine deployment would force exactly that regression on users who only need help navigating a form or a hearing. Domain adaptation is a trade, not a universal upgrade: the model’s decoder cross-attention shifted toward court-register morphology, which helps legal instruments but taxes open-ended dialogue.

Rule 2: if you are fine-tuning, stratify your corpus by the four Nepali legal registers — statute, judgment, contract, and treaty — and split dev/test to mirror those proportions. The submission reports that random splits overstate legal-domain BLEU by an average of 1.9 points across 5 seeds. A dev set with too many statute sentences will make your early stopping too optimistic and your legal test score too convenient to trust.

Rule 3: select checkpoints by legal-dev validation loss, never training loss. Roughly 120,000 updates into fine-tuning, training loss and dev loss diverge. Selecting on training loss underperforms dev-loss selection by 1.4 BLEU on the held-out legal test. Use early stopping on legal-dev loss and ignore the ascending training curve.

Rule 4: ship a checkpoint only when three metrics improve together on the dev set: sacreBLEU, chrF2, and LegalNER exact-match. If BLEU rises while chrF2 falls, the model is memorizing templates rather than generating acceptable morphology. That failure occurred in 2 of 7 fine-tuning runs in the submission, and the correct response is to roll back one epoch. The LegalNER 300-term list catches unseen terminology misses that BLEU hides, so it is the real guard against the “lexical memorization” failure mode.

Rule 5: for any production stream that is more than 30% non-legal text, put a fastText domain classifier before the MT engine, trained on 200k Nepali legal/non-legal sentences. Route legal sentences to the fine-tuned model and non-legal sentences to the generic NLLB checkpoint. The routed pipeline scores 30.9 BLEU on the 50/50 mixed test, versus 28.4 for fine-tuned-only and 26.1 for generic-only.

The decision tree is short. If the text is a statute, court judgment, or contract, deploy the fine-tuned checkpoint. If it is a legal-aid dialogue or conversational exchange, keep generic NLLB. If the incoming stream is more than 30% non-legal, add the fastText router. While fine-tuning, stratify by the four registers, optimize legal-dev validation loss, and require sacreBLEU, chrF2, and LegalNER exact-match to move upward together. When chrF2 drops, roll back one epoch. That sequence is the whole decision, and it is the only way to consistently get the domain gain without paying for it in mixed traffic.

What to do next

Frequently Asked Questions

What sacreBLEU does the full-weight fine-tuned NLLB-200-1.3B checkpoint achieve on the Edinburgh legal test set?

It achieves 32.6 sacreBLEU versus 23.4 for the generic NLLB-200-1.3B checkpoint, a statistically significant +9.2 gain.

How much of the fine-tuning gain is lost if the 2,800 case-heading templates are removed from training?

Removing all 2,800 case-heading templates lowers the gain by only 1.1 BLEU (32.6 to 31.5), so the gains are not lexical template memory.

What does the adapter-only ablation add over generic, and why is it insufficient?

Adapter-only (frozen encoder plus 32 rank-8 adapters) adds only +2.1 BLEU over generic (25.5), a small fraction of the full fine-tuning gain, because parameter-efficient shortcuts do not update decoder cross-attention layers 18–24 deeply enough.

How does the untuned 54B SMALL-100 compare with the fine-tuned 1.3B model on the legal test set?

SMALL-100 54B scores 28.9 BLEU untuned, while the fine-tuned 1.3B model beats it by 3.7 BLEU, showing capacity cannot substitute for register adaptation.

What is the exact BLEU gain and data cost of the backtranslation baseline in the ablation?

A 1M-pair backtranslation baseline adds only +1.4 BLEU over generic (24.8), while the 128,412-pair in-domain fine-tune adds +9.2, a 6.5x return on 13% of the data.

How does Google Translate's May 2026 production NMT score on the same legal test set?

Google Translate’s NMT scores 27.1 BLEU, 5.5 points below the fine-tuned open-weight model.

Quick answers

Sources: Co

Also worth reading: How to get the perfect AI summary of your PDF documents every time: How to get the perfect · AI-Powered OCR Enhances Ottoman Turkish to English Translation Accuracy: AI-Powered OCR Enhances Ottoman Turkish · AI-Powered Japanese-English Translation Accuracy Improvements in 2024: AI-Powered Japanese-English Translation Accuracy Improvements