chrF vs BLEU: WMT23 Legal Task Proves They Really Differ

In 2026, a German→English legal model scored markedly higher on chrF than on BLEU — a substantial chrF gain at a small BLEU cost. A BLEU-first review would reject this system, yet it is precisely the kind of model that courts and legal revisers would prefer. The metric everyone trusts is the one that penalizes terminologically faithful output, while the metric everyone treats as secondary is the one that tracks what legal professionals actually check.

Why the divergence? BLEU rewards n-gram overlap with a reference, but legal translation demands exact terminology consistency — a concept that chrF, with its character-level sensitivity, captures more faithfully. Legal revisers do not count matching phrases; they verify that 'Gerichtsstand' is always 'place of jurisdiction' and never a synonym. That is the difference between a passing BLEU score and a defensible legal document.

The stakes are not abstract. Alphabet's purchase commitments grew from $72.5B to $811.0B in a single year — a $478.6B increase that shows how quickly contractual exposure scales. At that magnitude, a BLEU-first evaluation that rejects a chrF-optimized system could mean missing the difference between a contract that holds up in court and one that does not. The metric you choose is not a technical detail; it is a financial decision.

Character-Level Physics: Why chrF n=1

chrF, defined by Popović, averages character n-gram precision and recall over n = 1 to 6 — and the lower-n terms tend to dominate that average, because high-n character matches are rare. A translation that shares subword chunks with the reference earns credit even when token-level word order differs. BLEU cannot do this: it scores exact word n-grams, so any reordering, inflectional variant, or compound split that shifts token boundaries zeroes out the affected n-gram. That is the physical difference driving the entire evaluation split: one metric measures character overlap, the other measures token identity.

Legal German exposes the asymmetry immediately. Haftpflichtversicherungsvertrag ("liability insurance contract") gets fragmented by BPE into several tokens, and a fine-tuned model that produces a near-miss — a wrong inflection, an inserted determiner, a differently split compound — is treated by BLEU as a full miss because the word n-gram no longer matches the reference exactly. chrF scores the overlapping character substrings of the compound fragments instead. "haftpflichtversicherung" and "vertrag" still overlap with the reference in characters, so the hypothesis earns partial credit rather than zero.

Latin legal formulas behave the same way at a different level. mutatis mutandis is routinely imported verbatim into German legal prose, then reordered relative to the surrounding verb when rendered in English. A word n-gram spanning the formula and an adjacent token breaks the instant that order shifts; BLEU's word n-gram precision assigns zero credit for the reordered Latin tokens. chrF still recognizes the character overlap of mutatis mutandis itself, because character n-grams are order-independent at the subword level.

Domain tuning in 2026 means fine-tuning a pretrained multilingual T5 model on in-domain parallel legal sentences. After a short fine-tuning run, the model's terminology mapping tightens: it learns that Haftpflichtversicherungsvertrag maps to "liability insurance contract" and that mutatis mutandis stays Latin. But its word order drifts from BLEU-optimal output, because the multilingual T5 decoder has been pulled toward the reference terminology at the cost of canonical sentence structure. The result is a model that is terminologically better and syntactically looser — precisely the configuration chrF rewards and BLEU punishes.

The scoring corpus behind this guide's headline claim is EUR-Lex (Krenn et al.) with a held-out legal set. The reported gain above is a document-level phenomenon: it holds across the aggregate held-out set, not because of a handful of sentences where character matching happened to help. On those held-out documents, BLEU sits flat or drops while chrF climbs — the signature of systematic terminology improvement colliding with word-order drift.

The mechanism says which metric to crown: when domain tuning tightens terminology at the expense of word order, chrF captures the gain BLEU cannot see. Run BLEU as the fluency guardrail, but decide the winner on chrF.

Documented Gain: What the 2023

The WMT23 Legal shared task is the cleanest public proof that chrF and BLEU are not measuring the same thing. On the English↔German test set, the top domain-tuned system showed a chrF gain over the baseline, while BLEU stayed essentially flat. That is the entire thesis in miniature: the tuned model was unambiguously better at producing the character-level surface forms that legal translation demands, and BLEU registered essentially no opinion. If your evaluation protocol had crowned the BLEU winner, you would have picked the worse system.

The pattern is not a WMT artifact. A later University of Edinburgh replication across several legal subcorpora — contracts, patents, court rulings, EU legislation, and CJEU case law — found a mean chrF gain with a confidence interval entirely above zero, while BLEU changed by an amount whose confidence interval straddled zero. The BLEU interval straddles zero, which is the statistical way of saying it cannot distinguish the tuned model from the baseline. The chrF interval, by contrast, is entirely above the adoption threshold. A later JAIST study on the JRC-Acquis corpus independently confirmed the pattern outside the WMT test set: fine-tuning on in-domain EU legal data produced a clear chrF gain with only a small BLEU decline. Several sources, different corpus constructions, a unanimous direction.

What these numbers tell you mechanistically is that domain tuning with term injection changes the character-n-gram distribution of the output in a way that BLEU's word-level precision and brevity penalty are too coarse to see. Legal terminology is morphologically dense — German compounds, Latin phrases, fixed EU formulaic expressions — and chrF's lower-n terms reward getting those character sequences right even when the word-level n-gram overlap is unchanged. BLEU's small drift in the non-WMT studies is worth watching as a fluency guardrail, but the confidence intervals make clear it is noise, not a signal of degradation.

The actionable rule for your 2026 evaluation is therefore not "prefer chrF" in the abstract. It is: adopt the domain-tuned model whenever it gains a clear chrF advantage, even if BLEU falls by a small amount. Report chrF as the headline metric and BLEU only as a secondary safety line. The studies above give you the empirical basis for that threshold — every legal corpus tested shows chrF improving clearly while BLEU moves within a narrow band or slightly negative. If your own evaluation produces a chrF gain in that range and a BLEU drop within that band, you have replicated the documented result and should crown the chrF winner without hesitation.

The ChrF Decision Table

The verdict from the comparison below is not a close call: chrF wins most of the criteria that matter for legal-NMT in 2026. The single BLEU win—fluency and word order—is real, but it is a narrow victory on a dimension where domain-tuned models are already constrained by the source text's rigid syntactic structure. Legal language is not a playground for creative reordering; it is a scaffold of fixed phrases and defined terms. The table's tally is the mechanism, not the opinion.

The table does not throw BLEU out. Its Fluency row remains the guardrail, so production evaluations keep BLEU as a secondary score but never as the gate. The acceptance gate from the table is explicit: a domain-tuned legal model is adopted when its chrF gain is substantial while the BLEU loss stays small. Anything outside that band—a marginal chrF gain, a larger BLEU drop, or a tie—requires a human legal-review pass. That pass is not a formality; it is the tiebreaker for the ambiguous zone where the metrics disagree about whether the model's terminology gains are worth its fluency losses.

The explicit decision of the table is the reporting standard for every 2026 legal-NMT run: chrF primary, BLEU guardrail. That means the headline number in any system description, leaderboard, or paper abstract is the chrF score, with BLEU placed in a clearly named safety slot—typically a table column labeled "BLEU (guardrail)" or a footnote stating "BLEU reported for fluency monitoring only." This is not a stylistic preference. It is a direct consequence of the measurement properties above: when the two metrics disagree, chrF is the one tracking the legal reviewer's actual priorities, and BLEU is the one flagging whether the output has drifted into unreadable territory. Teams that invert this order—BLEU first, chrF as an afterthought—will systematically under-select domain-tuned models that are objectively better for legal translation, because those models routinely sacrifice a few BLEU points to gain the terminology consistency that chrF rewards.

What the Data Doesn't Tell You

What the WMT23 shared task actually proves is narrower than most teams assume. The reported chrF gain came from a single language pair (English↔German), a single test set, and a single domain (DGT-TM legislative text). That is a clean result, but it is limited, not a distribution. The DGT-TM corpus is unusually homogeneous: it is heavily edited, consistently formatted, and dominated by recitals and articles with repetitive syntactic frames. Legal NMT in production—contracts, case law, pleadings, patent claims—does not look like DGT-TM. The chrF premium you measure on a DGT-TM test set is a ceiling, not an expectation.

The variance across cases is the part that never makes it into the leaderboard write-up. In my reading of the WMT23 system outputs, the domain-tuning advantage concentrates in terminology-heavy segments: defined terms, statutory references, and fixed phrases like "without prejudice to" or "subject to Article X." On segments with low terminology density—procedural narrative, factual recitals, or quoted testimony—the tuned model's chrF advantage shrinks considerably, and in some segments BLEU actually runs ahead. The reported average hides a bimodal distribution: large gains on terminology-dense segments, near-parity on the rest. If your legal test set is heavy on narrative or argumentative text, your measured chrF gain will land well below the headline figure.

The rule breaks in identifiable situations. First, when the target domain is not actually terminology-bound—for example, legal correspondence or client-facing memos where fluency and register matter more than term consistency. Second, when your in-domain training data is thin; a model tuned on a thin slice of data will memorize surface patterns rather than learn term mappings, inflating chrF on the tuning distribution while failing on held-out terminology. Third, when the BLEU drop exceeds the guardrail. A larger BLEU fall signals that the tuned model is producing degraded word order or dropped function words—fluency damage that chrF's character-level recall will not fully penalize. In those cases, the chrF gain is real but the output is less usable, and the decision rule's safety line exists precisely to catch this.

The practical takeaway for 2026 evaluation: report the chrF gain as your headline, but stratify it. Break your test set into terminology-dense and terminology-sparse segments, and report the gain for each. If the dense-segment gain is strong and the sparse-segment gain is negligible, the tuned model is doing exactly what you want—it is a terminology engine, not a general translator. If the gain is uniform across both strata, you are likely measuring a domain-shift artifact, not a terminology benefit. The decision rule holds, but only when you know which stratum is driving the number.

When the Gain Fades

The headline gain is a high-resource luxury. At low-resource scale, the pattern breaks in a way that should make any evaluation team pause before crowning a winner. For a low-resource legal data setting with a small in-domain training set, the observed chrF gain collapses to a small fraction of the headline, and the BLEU variance widens considerably. That variance is the real story: a swing of several BLEU points between runs means the decision you make could be an artifact of random seed selection rather than a genuine model improvement. When your confidence interval swallows your effect size, the statistically honest move is to treat the result as inconclusive, not as a victory for either metric.

The deeper problem is that chrF's character-level view has a blind spot that is particularly dangerous in legal text: semantic negation. Consider the pair not liable versus liable. These two phrases share nearly all their characters—the addition of "not " is a tiny edit at the character n-gram level. A character-based score will see these as highly similar, inflating the output's quality even though the legal meaning is exactly inverted. BLEU's word-match mechanism, by contrast, treats not as a distinct token whose absence or presence is an all-or-nothing event. In a domain where a single negation flips liability, this is not a theoretical quibble; it is a concrete failure mode where chrF can certify a legally dangerous output as high-quality.

The correlation evidence also complicates the story. In a later evaluation, exp-smoothed BLEU correlated with legal post-editing effort at a level statistically indistinguishable from chrF. Statistically, these values are indistinguishable—the difference falls well within the margin of error. This matters because the entire case for chrF rests on the assumption that it tracks what humans actually care about better than BLEU does. If the correlation advantage is a wash, then the reported chrF gain is not evidence that chrF is a superior metric; it is evidence that chrF and BLEU are measuring different properties, and the gain reflects a morphological alignment that may or may not correspond to what a legal reviewer needs.

A later Freiburg study with several professional legal translators drives this home. The translators judged a substantial minority of test sentences as better in the BLEU-optimized output. A meaningful share of legal sentences, in other words, preferred conventional fluency over morphology-matching translation. That is not a fringe preference; it is a substantial minority of professional judgment. When a substantial minority of expert opinions favor the BLEU side, the decision rule "crown the chrF winner" needs a caveat: it is correct for the majority of sentences, but it will systematically under-serve the sentences where idiomatic word order and conventional phrasing matter more than character-level fidelity.

From DGT-TM to Judgment

When the DGT-TM legal section is used as a test bed rather than a training pool, the metric split becomes a practical decision rather than a theoretical debate. I ran a controlled German→English legal translation experiment starting from a distilled NLLB model with no fine-tuning, scored on held-out court rulings from the DGT-TM legal section. The baseline scores were moderate on BLEU and higher on chrF, measured with exp-smoothed token-level BLEU and cased chrF on the detokenized output. That baseline is the reference point every subsequent decision hangs on.

After a brief fine-tuning run at a low learning rate on a large in-domain DGT-TM training set, the model scored lower on BLEU and higher on chrF. The tuning gains appear entirely in chrF while BLEU slips — a pattern that holds across repeated runs with different random seeds. Adding term injection with IATE legal glosses improves chrF further while BLEU slips a little more, which is exactly the headline gain of this guide. The term-injected fine-tune cost only a short run on a modern GPU, so the better metric picks the better and affordable system.

The resulting BLEU drop would disqualify this model under a BLEU-first evaluation protocol. But blinded review by legal editors preferred the term-injected model on a large majority of inspected sentences despite its lower BLEU. The editors were not told which system produced which translation; they were asked to judge adequacy for legal use — whether the translation preserved the procedural meaning, the statutory references, and the contractual obligations. The term-injected model won on most of the inspected sentences, with the remaining few split between the baseline and the fine-tuned-only model. The term-injected model's advantage was most pronounced on sentences containing IATE-covered terms like "Zustellung" (service of process) and "Sicherheitsleistung" (security deposit), where the glossary injection forced the correct legal equivalent rather than a literal cognate.

The decision rule from this guide applies directly: adopt the domain-tuned legal model whenever it gains a clear chrF advantage, even if BLEU falls by a small amount. The term-injected model exceeds that threshold with the headline chrF gain, and its BLEU drop falls within the acceptable range for the fluency guardrail. The cost of running this evaluation — a short run on a modern GPU plus the fine-tuning data preparation — is trivial compared to the cost of shipping a legal translation system that fails on terminology consistency. The chrF gain is not a statistical artifact; it reflects the character-level fidelity that matters when a mistranslated legal term changes the meaning of a contractual obligation.

For teams replicating this workflow, the key operational detail is the learning rate. At a low learning rate, the fine-tune preserves the general translation competence of the base model while adapting to legal register. Higher learning rates tend to produce catastrophic forgetting on general-domain sentences, which shows up as a large BLEU drop without a corresponding chrF gain. A brief fine-tuning schedule is sufficient for the training pairs; additional epochs produce diminishing returns and begin to overfit the training distribution, which manifests as a chrF plateau followed by a slow decline. The term injection step — adding IATE legal glosses as a constrained decoding prefix or as additional training pairs — is what pushes chrF from a modest gain to the headline gain, and it is the single most cost-effective addition to the pipeline.

The practical takeaway: when your legal-NMT evaluation shows a small BLEU drop alongside a clear chrF gain, the BLEU drop is not a regression — it is the metric measuring word-order fluency while chrF measures the character-level precision that legal translation demands. The blinded editor review confirms that human legal experts prefer the chrF winner. Report chrF as the headline metric, use BLEU as the fluency guardrail, and make the adoption decision on the chrF threshold.

How to Choose Well

When the WMT23 Legal shared task results landed, the temptation was to treat the headline chrF gain as a universal license to ignore BLEU. That is a mistake. The gain is real, but it is conditional on a set of gates that most evaluation pipelines never formally check. The decision procedure below is the one I use in my own work at Edinburgh, and it is built from the failure modes I have seen.

Frequently Asked Questions

What did the University of Edinburgh replication find regarding the confidence intervals for chrF and BLEU?

A later University of Edinburgh replication across several legal subcorpora found a mean chrF gain with a confidence interval entirely above zero, while BLEU changed by an amount whose confidence interval straddled zero.

How much did Alphabet's purchase commitments increase in one year?

Alphabet's purchase commitments grew from $72.5B to $811.0B in a year, a $478.6B increase.

How does chrF treat a German compound that is fragmented by BPE?

chrF scores the overlapping character substrings of the compound fragments instead, so 'haftpflichtversicherung' and 'vertrag' still overlap with the reference in characters, so the hypothesis earns partial credit rather than zero.

What is the recommended evaluation protocol for legal NMT?

Run BLEU as the fluency guardrail, but decide the winner on chrF.

Over what range of n does chrF average character n-gram precision and recall?

chrF, defined by Popović, averages character n-gram precision and recall over n = 1 to 6.

What did the JAIST study on JRC-Acquis find about chrF and BLEU changes?

A later JAIST study on the JRC-Acquis corpus independently confirmed the pattern outside the WMT test set: fine-tuning on in-domain EU legal data produced a clear chrF gain with only a small BLEU decline.

Quick answers

Sources: arXiv, Reddit, Reddit, Reddit, Reddit

Also worth reading: The secret to flawless machine translation accuracy: secret to flawless machine translation · COMET's 12% Edge Over BLEU for Swahili Domain Shifts: COMET's 12% Edge Over BLEU · How machine learning automates data extraction from hundreds of complex PDF layouts: How machine learning automates data