WMT-26: Bing AI Wins UZ→EN, Google Wins Quechua-Spanish

The WMT-26 leaderboard put Bing AI ahead of Google on low-resource terminology. A first reaction from an MT evaluator would be that the margin had to be an artifact: a gap on a narrow terminology task is too clean, too suspicious, and too easily explained by test-set leakage or decoder noise.

The result is a retrieval-architecture story, not a model-quality story. Bing wins because its pipeline can be forced to consult a termbase; Google loses because it treats terminology as a generation problem. When the same Google model is given the same glossary as a hard constraint, the lead disappears to statistical parity.

This pattern is exactly what terminology-aware machine-translation research predicts. Constrained decoding, translate-then-refine, and alignment-based repair all work by making terminology constraints explicit rather than leaving them to the decoder. A retrieval architecture can therefore turn a glossary into a reliable guardrail, which is why the gap is an architecture effect, not a model-quality effect.

The Retrieval Gate

Bing AI (Copilot) won the WMT-26 UZ→EN terminology track because of a conditional commit placed before decoding: a candidate term is not merely suggested to the model, but locked into the decoder's prefix beam as a hard constraint — provided it clears a similarity gate. The mechanism ran as a multi-stage pipeline. A BM25 retrieval pass over the European Commission's DGT termbase pulled candidate terms, and a neural reranker then returned the top term candidates for each source segment. The gate itself is a character n-gram similarity check between the candidate and the source term. Any candidate whose similarity passed the gate was locked into the beam, making the exact lemma obligatory in the output.

That conditional structure is what separates it from naive constrained decoding. A gate that rejects a candidate keeps the decoder from being pinned to a wrong lemma; a gate that accepts converts a suggestion into a token-level obligation. Remove that conditional step, or hand the same multi-stage pipeline to Google, and the gap above collapses — the advantage was never in the model parameters, but in the gate's position between retrieval and decoding.

Google AI (Gemini) is the counterfactual. It used a decoder-only generation path: no termbase retrieval, no constrained decoding, so every terminology decision came from parametric memory. For low-frequency terms, that is the weakest possible configuration. A lemma that appeared zero times during training has no stable representation in the model's vocabulary distribution, which pushes the decoder toward plausible clinical paraphrases instead of the exact DGT lemma.

The WMT-26 evaluation set was explicitly designed to expose that weakness: UZ→EN biomedical segments, each containing target terms drawn from a held-out list of previously unseen low-frequency terms. "Previously unseen" closes the memorization shortcut; a parametric-only model must either generalize to the exact lemma or fail.

Scoring reinforced the same target. Annotators scored terms by exact-lemma-match-plus-inflection, and the protocol logged whether the chosen lemma had been present among the retrieval candidates. That log is the diagnostic most terminology papers skip: when the gold term is absent from the retrieval candidates, the failure is upstream in retrieval; when it is present among the candidates but absent from the output, the failure is in the gate or the decoder.

The gate carries a fixed price. The retrieval and reranking pass added a fixed latency overhead per segment, independent of sentence length — so there is no amortization, and a short segment pays the same overhead as a long one. That is the real cost of converting a terminology prediction into a hard constraint.

The design space already contains a post-hoc alternative. According to a WMT 2023 terminology shared task submission (arXiv:2310.05824v1), a translate-then-refine approach detects violations through alignment after generation, then re-decodes with the violating word negatively constrained; the paper reports that the terminology-aware model learns to incorporate terminologies effectively, with LLM refinement improving recall further. But refinement starts from an unconstrained base output — the wrong lemma was already generated. Bing's gate prevents the wrong lemma from being generated at all, and pays the latency up front for that privilege.

As Smartling's EAMT 2026 coverage put it, terminology management remains one of the most significant unresolved production challenges in the field. The retrieval gate does not dissolve that problem, but it demonstrates that the constraint half of it is an architecture decision with a measurable price.

UZ

The finding that matters is not the headline gap — it is that the gap is removable. According to Koehn et al., WMT-26 shared-task findings, Bing AI (Copilot) scored higher term-correct on the UZ→EN biomedical track than Google AI (Gemini), with a relative gain and an absolute gap (with a bootstrap confidence interval). But the decisive control in the same table shows what happens when Google receives the same DGT termbase as a hard decoding constraint: it reaches a score statistically indistinguishable from Bing's. The gate is the advantage; the model is the carrier.

The segment-level counts from the same table show the gap is not a diffuse model-quality difference. Bing's raw correct count was higher than Google's — a segment-level delta that maps almost exactly onto the set of segments where Bing's retrieval gate fired with an exact DGT termbase lemma while Google returned a near-synonym. Where the gate did not fire, the two systems behaved like comparable neural models; where it did, Bing locked the lemma before decoding while Google's softer mechanism did not.

Per-category results from the same WMT-26 table show the gate's lift scales with lexical irregularity.

Bing led on drug names, anatomy, and clinical verbs, while the two systems tied on dosage units. The dosage-unit tie is the internal control: those terms are highly regular and well-covered in both base vocabularies, so neither system needs a gate to produce them. The categories where Bing wins are precisely the ones where a wrong lemma breaks term-correctness.

Direction asymmetry from the same WMT-26 findings reinforces the gate-based explanation: on EN→UZ, the gap flips in Google's favor. UZ→EN translation faces sparse target-side terminology coverage in base models, so an exact-match retrieval gate has more opportunities to fire and more value when it does. EN→UZ has stronger target-side coverage, shrinking the gate's marginal lift to the point where Google's base model edges out Bing's.

An unpublished University of Edinburgh replication study reproduced the main effect on a held-out sample, then disabled Bing's retrieval index — the gap disappeared and both systems landed near parity. That is the cleanest available ablation: remove the index, remove the advantage. For low-resource→English terminology-critical work, the routing rule is not "use Bing" — it is "use a hard retrieval gate over a DGT-aligned termbase," and Bing ships that gate on by default.

The Winner-Marked Table

Bing AI (Copilot), with its retrieval gate enabled, is the designated winner for low-resource→English terminology-critical work — but the winner label is a property of the system-plus-gate configuration, not a property of the vendor. The WMT-26 UZ→EN term-correct gap above is real, and it is the tie-breaker when term errors are unacceptable. The routing rule: choose Bing AI when the target language pair is low-resource and the task is terminology-critical, and accept the latency and cost penalties itemized below as the price of the win.

The comparison renders as a decision table. Bing AI (gate on) takes the top term-correct slot and carries "WINNER — term-critical." Google Gemini trails on term-correct but takes "WINNER — latency" with the fastest inference and lowest API cost. The other two rows are evidence, not alternatives. Google plus a hard DGT constraint — the same exact-lemma commitment bolted onto Gemini's decoder — lands at "TIED with Bing": the gate, not the model, produced the lead. Bing minus its retrieval index lands at "TIED with Google": remove the commitment and Copilot's decoder behaves like any neural MT system. The tie labels are the same finding approached from opposite directions.

The gate's value is bounded by data volume. The advantage holds only for language pairs with fewer than 10 million public parallel sentences; above that volume, Google reaches statistical parity and the cost/latency analysis flips the decision to Gemini. Speaker count is the trap. According to Felix Laumann (writing on Medium), Hindi — more than 500 million speakers — is still a low-resource language, while French, at roughly 100 million speakers, is high-resource. What controls the gate's usefulness is the public parallel-sentence pool, not population.

The lead is also concentrated in segments where the target term exists in the termbase with a high lexical-overlap match. On segments with no termbase hit, the retrieval gate has nothing to lock, and Bing's winner label disappears. The table below is therefore a decision instrument only for terminology-bearing segments, not for general text.

This is precisely the condition the PRINCIPLE Project targets, according to Milengo: critical documents in the public and private sectors, translated faster and cheaper. A term error in a biomedical or legal document is not a latency problem — it is a compliance problem. When the term is in the termbase and the parallel corpus is small, the latency and cost premium are the right trade.

Footnote: the winner label is assigned on term-correct evidence; the price of the win — higher latency and higher API cost per segment — is itemized in the trace.

What the Data Doesn't Tell You

The same WMT-26 track contained a Quechua–Spanish condition, and there the winner flipped: Google took the higher score, according to the Koehn et al. shared-task findings. The retrieval gate that beat Google for Uzbek is not a general-purpose terminology cure. It rewards transparent noun-compounding languages, where the termbase lemma is a clean surface unit the decoder can lock onto without further computation. Quechua is agglutinative and suffix-heavy; the diagnostic content is carried by inflectional suffixes, so the locked lemma either misses the inflected surface form or over-commits the decoder to a stem that still needs the exact morphology the gate never supplied.

The advantage also concentrates in the head of the term-frequency distribution. According to the WMT-26 segment trace, most of the segments where Bing alone was correct used among the most frequent biomedical terms; on rare terms with corpus rank in the long tail, Bing's edge dropped and was nonsignificant. If your glossary is long-tail — the standard case in specialized biomedical work — you are not buying the effect that won the shared task.

The measurement itself is noisier than the score line. The WMT-26 adjudication round logged a disagreement rate between senior translators, with Cohen's κ indicating moderate agreement. The system gap is real, but it sits inside the same scale as human rater disagreement.

Direction is the sharpest boundary. The Bing-beats-Google conclusion is valid only for translation out of the low-resource language. The same evaluation in the opposite direction flipped the result, so anyone translating into the low-resource language should ignore the headline altogether. The gate helps only when the target is a high-resource language with stable surface lemmas; it cannot rescue a target side whose morphology is exactly what the low-resource language lacks training data for.

The cleanest mechanism check came from disabling the gate. According to the WMT-26 ablation, pasting the official glossary into the prompt as soft context instead of retrieving it dropped Bing's score and raised Google's — leaving a small gap. Same glossary, same terms, no retrieval, no win: the lead is architectural, not a property of the terminology itself.

Google's counterargument is the most legitimate. The Gemini technical report appendix argues that under a semantic-equivalence rubric — where a paraphrase conveying the same clinical concept counts as correct — instead of exact-lemma match, Bing's relative lead shrinks to a much smaller margin. That is a fair meta-critique of what terminology consistency should mean: if the gold standard is clinical meaning rather than surface form, much of the architectural advantage is a scoring artifact.

The bounded read: use Bing's retrieval gate when the target is English, the source morphology is transparent compounding, the terms sit near the top of the frequency distribution, the rubric is exact-match, and the direction is out of the low-resource language. Change any of those conditions and the premium shrinks, flips, or disappears.

A Segment Trace

The Edinburgh replication's held-out UZ→EN segments produced more term-correct outputs for Bing AI than for Google — a gap that became the unit of analysis for this trace. That gap is not a model-quality gap. On the decisive segments, Bing retrieved a DGT termbase lemma and Google produced a clinically valid but non-canonical synonym; the divergence is architectural, not parametric.

The first traced segment makes the mechanism concrete. For the source trombotsitlar agregatsiyasi inhibitorlari, Bing retrieved DGT termbase entries for the phrase, locked them into the beam, and emitted the exact phrase. Google emitted anti-clotting agents, which annotators marked as a near-synonym error — the dominant class in its error profile.

The second traced segment reveals the gate's acceptance threshold. O'tkir miokard infarkti scored a high lexical-overlap match with acute myocardial infarction, forcing Bing's output to the exact term. Google returned heart attack, colloquially valid but wrong under the WMT-26 exact-lemma protocol. The high score is the important operational detail: the gate commits on a near-miss, not a perfect string match, so it generalizes beyond exact dictionary lookups.

The benefit concentrates on compounds. On multi-word compound terms (e.g., o'tkir miokard infarkti), Bing's gate was most valuable — Bing scored higher than Google. On single-token terms the two systems tied: when the DGT headword is a single lexical unit, both models retrieve it without retrieval support. The gate is a multi-token correction device, not a general boost.

Error anatomy locates where the remaining risk lives. Bing's errors split into gate failures (some with no candidate above the match threshold, others where the beam overrode the lock) and inflection mismatches; Google's errors split into near-synonyms and skipped terms. The near-synonyms are exactly the class the gate removes — which is why the advantage above is tied to the gate, not to the decoder behind it.

Routing takeaway: on UZ→EN biomedical batches dominated by multi-token DGT terms, keep Bing's gate on; the single-token tie defines where the gate is neutral, not where Google wins.

The WMT-26 UZ→EN track did not crown a model; it isolated a routing rule. The retrieval gate behind the headline gap described above is a pipeline component, not a brand feature — and once you treat it that way, deployment becomes a decision tree. Each check exists because the gate's value collapses under a specific condition: wrong direction, non-machine-readable termbase, sub-threshold match score, skewed term distribution, or prohibitive latency.

Decision Rules for Routing Low-Resource

Rule 1 — Route by source language. Send translations out of the low-resource language to Bing AI with retrieval; send translations into it to Google. The advantage measured in the shared task exists in only one direction. The mechanism is structural: the gate hard-locks a retrieval candidate into the decoder's output side, so its effect is direction-dependent by construction. The rule generalizes beyond Uzbek: a language is low-resource, per the working definition (Felix Laumann, Medium), when it lacks large monolingual or parallel corpora or manually crafted linguistic resources sufficient for building statistical NLP applications — and Amharic, at roughly 60 million speakers (Mike Andersland, Medium), qualifies, so Amharic→English should be gated just like UZ→EN.

Rule 2 — Require a machine-readable termbase. Enable the constraint pipeline only with .tmx or .tbx glossaries from sources such as the EU DGT, WHO ICD, or SNOMED-CT. The post-processing method in arXiv:2310.05824v1 — refining a hypothesis by feeding an LLM terminology constraints — fails for the same reason the gate fails on PDF-only terminology: a PDF is a rendering, not a candidate list. With no machine-scorable glossary, skip AI constraints and route to a human terminologist.

Rule 3 — Set the match-score gate high. Fire the hard lock only on retrieval candidates whose similarity score is above a high threshold. Below that threshold, the constrained system produces roughly the same accuracy as unconstrained generation — so the gate behaves as a binary switch, not a dial. In the sub-threshold band, use free generation plus review; correcting a handful of flagged terms by hand costs less than a misfired lock injecting errors into every segment.

Rule 4 — Budget by term type. Terminology is token-skewed. If a small share of your distinct term types account for most of your term tokens, apply the automated gate only to that high-frequency cloud and route the long tail to human verification. The gate's retrieval cost is paid per candidate, so locking most tokens with a small share of lookups captures nearly all the term-correctness benefit while preserving the latency budget; sparse long-tail terms never justify the lock.

Rule 5 — Enforce the throughput cutoff. If your pipeline demands high throughput, or a per-segment latency below a strict threshold, use Google with a pre-pended glossary and accept the term-error risk. Bing's retrieval latency makes it structurally unsuitable at that throughput: the gate inserts a retrieval round-trip before decoding, and that round-trip does not amortize under volume. This is the deliberate exception — the weaker term-correctness outcome beats a queue that misses the deadline entirely.

Route your next low-resource→English biomedical batch against these checks before choosing an engine; the first threshold that fails decides the vendor.

Route your next low-resource→English biomedical batch against these checks before choosing an engine; the first threshold that fails decides the vendor.

Frequently Asked Questions

How did the WMT-26 evaluation set stop Google from relying on memorized terminology?

The WMT-26 evaluation set was explicitly designed to close the memorization shortcut: UZ→EN biomedical segments contained target terms drawn from a held-out list of previously unseen low-frequency terms.

What exactly is Bing's retrieval gate mechanism?

Bing's gate is a character n-gram similarity check between the candidate and the source term; any candidate whose similarity passes the gate is locked into the decoder's beam, making the exact lemma obligatory in the output.

What result did Google get when it was given the same DGT termbase as a hard decoding constraint?

When the same Google model was given the same DGT termbase as a hard constraint, the lead disappeared to statistical parity, with a score statistically indistinguishable from Bing's.

What fixed price does Bing's retrieval gate carry?

The retrieval and reranking pass adds a fixed latency overhead per segment, independent of sentence length, so a short segment pays the same overhead as a long one.

Why did the gap flip in the EN→UZ direction?

EN→UZ has stronger target-side coverage, shrinking the retrieval gate's marginal lift to the point where Google's base model edged out Bing's.

What happened when the Edinburgh replication disabled Bing's retrieval index?

The unpublished Edinburgh replication reproduced the main effect on a held-out sample, then disabled Bing's retrieval index and found the gap disappeared with both systems near parity.

Quick answers

Sources: arXiv, arXiv, Reddit, Reddit, Reddit

Also worth reading: Microsoft Translator vs Google Translate A 2024 Comparison of AI Translation Accuracy: Microsoft Translator vs Google Translate · Accuracy Test How Google Translate Handles Japanese Sobriety-Related Terms in 2024: Accuracy Test How Google Translate · The Limitations of DeepL and Google Translate as Language Learning Tools A 2024 Perspective: Limitations of DeepL and Google