Translation quality check: 1st diagnostic—character-level F-score (chrF), not release queue

The LoResLM 2025 overview reports 35 accepted papers from 52 submissions, spanning 8 language families and 13 research areas. Those figures signal workshop breadth, not translation performance: the source supplies no systems, datasets, train/test splits, translation error rate, or quality score. A published average error reduction therefore cannot establish that a model is ready for a 2026 release.

chrF is a character-level diagnostic, not a release queue. The defensible selection order is lexicographic: eliminate failures involving critical terminology, named entities, and required casing; only then compare case-insensitive error rates and blinded adequacy among survivors. Case-insensitive matching must apply Unicode casefolding rather than an unspecified conversion, with equality or containment selected explicitly.

The supplied LoResLM abstract cannot validate that framework. Submitted on 20 December 2024 and focused on a 2025 workshop, it contains no 2026 experiment and names no models, systems, datasets, splits, or low-resource languages for evaluation. Its participation counts should not be recycled as accuracy claims. For a 2026 decision, aggregate diagnostics come after hard terminology, entity, and casing gates, while blinded human adequacy checks test whether surviving candidates are usable.

Why chrF Orders 1

chrF belongs first in the diagnostic queue, not the release queue. According to Popović’s definition, chrF aggregates character n-gram overlap for n=1 through 6. I compute it on a Unicode-normalized, case-folded scoring copy and never overwrite the stored translation. The result is a continuous measure of overlap, not adequacy: a fluent sequence can preserve the wrong terminology or factual relation. A favorable chrF movement therefore cannot, by itself, establish deployment readiness.

In my low-resource NMT evaluation work, the useful causal model combines multilingual transfer with back-translation. Related languages share representations that can make supervision transferable; monolingual text is then converted into synthetic bitext. When authentic in-domain pairs are scarce, however, a teacher system’s terminology mistakes can become synthetic training targets and be reinforced. chrF may reward similarity to those teacher choices, allowing lexical overlap to improve while factual precision deteriorates.

The CI non-critical sentence-error rate is the number of adjudicated segments containing at least one non-critical content mismatch divided by all adjudicated segments. A segment counts once, however many non-critical mismatches it contains. Create the adjudication copies through Unicode normalization followed by full case folding, but retain diacritics, numerals, punctuation, and negation while experts determine meaning. Keep critical classes separate: a case-insensitive string match must not launder a critical defect into the non-critical category.

Before evaluation, predeclare protected terminology, including inflected and multiword forms. Compare morphological variants deliberately, but do not let normalization erase meaning or required case. Any wrong term sense, entity, numeral, negation, or required case makes the segment a critical failure, even if sentence-level chrF improves. Predeclaration prevents convenient post hoc relabeling and makes the relationship testable on held-out language–domain pairs.

On each held-out pair, bilingual domain experts blinded to system identity should score adequacy and error category. Report the median and rater spread rather than allowing one fluent but factually wrong output to pass on aggregate lexical overlap. The safer-deployment inference survives only when lower CI non-critical sentence-error rate accompanies no critical failures and stronger blinded median adequacy; a weaker median or any critical failure falsifies that inference for the pair.

Run a second, exact-case audit on the original output for code, tables, bibliographies, and regulated terminology. Casefold only the scoring copy: lowercasing a required field such as “PatientID” can make an otherwise usable translation operationally invalid. For a 2026 release, the canonical error rule—not chrF—controls the operational decision.

Scale Is Not Safety

CCNet makes the distinction between scale and safety explicit: its global total describes how much text was mined, not whether individual low-resource languages received balanced supervision. I treat multilingual corpus size as a hypothesis generator, then disaggregate it before drawing any conclusion about a specific translation direction or specialized domain.

The relevant unit is the exact language pair, target domain, and held-out split. Pooled totals can conceal severe sparsity in one language combination, while web-derived counts can obscure whether the available text resembles the terminology and prose of the intended deployment. Corpus statistics therefore identify where blind evaluation is urgent; they cannot substitute for it.

Average improvements on broad benchmarks also do not transfer automatically to specialized use. A model can benefit from abundant data in one direction while remaining unsafe where terminology is sparse or expert adequacy is weak. FLORES offers a useful common denominator, but comparability on general sentences does not establish terminology performance in a new domain.

For an unedited release, the aggregate result must be interpreted together with the article’s stated non-critical error threshold, zero critical terminology, entity, numeral, and required-casing errors, and stronger blinded domain-expert adequacy. Otherwise, use terminology-constrained MT with targeted human review. This relationship is falsifiable on each held-out language–domain pair: if aggregate error falls while any critical failure remains or expert adequacy does not improve, the purported safety gain fails.

The Lexicographic Scorecard

A scorecard should operate as a veto ladder, not a leaderboard. Fix the field order before inspecting candidate outputs: critical error count, with terminology, entity, numeral, and case failures retained separately; held-out case-insensitive (CI) non-critical sentence-error rate; median blinded domain-expert adequacy on a 1–5 scale; and then peak memory or latency. Evaluation stops at the first failed gate. A lower-priority score cannot compensate for a higher-priority failure, and neither adequacy nor latency can pardon a critical error. This explicitly kills the myth that a lower case-insensitive aggregate score automatically establishes translation quality or deployment readiness.

The following is a policy illustration, not a reported experiment. It uses a stipulated held-out set of segments. Before publication, every illustrated outcome must be replaced by a measured result on that frozen set, including the currently unspecified resource measurements.

Read the comparison lexicographically, not by sorting the aggregate-error column. The general-purpose LLM has lower aggregate error and higher adequacy, but its critical-error count fails the first gate. Terminology-constrained NMT becomes the explicit winner because zero critical errors cannot be offset by weaker lower-priority results. A weighted composite would allow those gains to conceal unacceptable deployment risk. “Winner” identifies the preferred architecture, not permission to bypass the stated release rule: raw candidates that fail it move to terminology-constrained generation with targeted human review.

Make every verdict reproducible by placing the model checkpoint, protected-term-list version, decoding settings, document split, and confidence interval beside every score. Rate intervals should be clustered by document because several segments can originate from one document; memory and latency measurements require the same execution metadata. A corpus average without those fields cannot adjudicate a launch.

Freeze a document-disjoint evaluation containing protected terms, numerals, negation, morphology, and code-switching. Report every domain slice separately, retain the critical-error categories, and predeclare comparisons before inspecting results. Otherwise, frequent easy material can make a system appear safer than it is on specialist language–domain pairs. Repeating the protocol on held-out language–domain pairs makes the thesis falsifiable: failure of the zero-critical and blinded-adequacy ordering rejects the claim rather than being rescued by a corpus average.

Implement the winning architecture as terminology-constrained generation for routine traffic. Escalate low-confidence outputs and protected-term matches for targeted human review. Retain raw and constrained outputs as aligned pairs with the constraint version and routing decision, so every terminology change remains auditable. The next release review should therefore begin with category-level critical counts, not the corpus leaderboard.

What the Data Doesn't Tell You

A lower aggregate error rate shows where errors are frequent, not which failures can change a deployment outcome. Consider the counter-case: candidate A has 1.0 percentage point more non-critical error than candidate B, yet A has zero protected-term or entity errors, while B makes one critical omission. A can therefore be safer despite its worse aggregate; the aggregate simply does not expose the severity tail. The lower-score advantage becomes a safety prediction only when critical terminology, entity, numeral, and required-case counts are all zero and blinded domain experts judge adequacy more strongly. That is a conditional empirical claim, not an automatic property of case-insensitive scoring.

Case-insensitive comparison creates its own blind spot. Unicode casefolding can map operationally distinct strings such as en-US and en-us to the same comparison key, while regular expressions, code identifiers, and citation keys may encode case as syntax or identity. A defensible audit therefore keeps the raw output and a casefolded comparison key separate: use Unicode casefolding for the requested pairwise, case-insensitive comparison, but validate protected targets again against their exact required forms. According to Unicode UTS #18, Version 25, regular-expression engines require explicit Unicode-aware adaptation; Unicode support does not justify erasing source case before exact validation. If the casefold pass succeeds while exact-case validation fails, that failure is counter-evidence against relying on the case-insensitive score alone.

COMET and BLEURT do not provide an independent terminology oracle. Both remain sensitive to the bilingual reference: when that reference substitutes the wrong technical term, a candidate reproducing the same substitution can receive a higher learned-metric score because it agrees more closely with the reference. Such movement reflects agreement with annotation error, not safer machine translation. Keep terminology adjudication outside the reference-based metric, checking the protected glossary and source document directly. Do not allow a COMET or BLEURT gain to cancel a critical terminology count; otherwise, a high semantic-quality proxy can launder a domain-critical error.

Human ratings introduce variance, not an automatic tie-breaker. With two raters using a 5-point adequacy scale, material pairwise disagreement should trigger a third bilingual adjudicator and a report of pre-adjudication agreement. Preserve both original ratings so that disagreement remains visible; then use the adjudicated result to assess whether one candidate has stronger blinded domain-expert adequacy. Until that step is complete, a rating gap can reflect rater variance rather than stable quality. Expert evidence strengthens the safety relationship only when it is independent of reference-based scores and the protected-error checks remain veto conditions.

A multilingual ranking is not time-invariant. A multilingual result from 2024 can reverse on 2026 specialized documents after terminology or domain mix changes, even if the evaluation methodology is unchanged. Report year-by-domain variance, including the direction and magnitude of rank changes, rather than pooling unlike strata into one reassuring average. If candidate order flips across language–domain strata—or across held-out language–domain pairs designed to test the relationship—do not name a single winner. The claimed safety premium is falsified for that setting when a higher-aggregate candidate has the clean critical-error record and stronger blinded adequacy while the lower-aggregate candidate does not. In that stratum, withhold aggregate-based clearance and use terminology-constrained MT with targeted human review.

A Deployment-Level Audit

Aggregate improvement can leave a system ineligible for release. A worked audit would require a measured baseline, a declared critical-error rule, and a verified result for the intended held-out language–domain case. The supplied LoResLM abstract contains no 2026 experiment and cannot substantiate a quantified release claim.

The pre-adaptation burden must be calculated from measured held-out results before testing whether an apparent improvement survives deployment-level scrutiny.

Any residual-error projection based on an assumed transfer of an aggregate reduction remains hypothetical until it is verified on the held-out language–domain case. Even a favorable projection is not permission to ship.

Bilingual adjudication can identify protected-term substitutions even when an aggregate projection improves. Any protected-term substitution is sufficient to fail the zero-critical-error condition, so the raw system cannot ship merely because its aggregate projection is favorable. This also kills the shortcut that a lower case-insensitive quality measure automatically certifies deployment readiness: the critical veto remains lexically prior to the aggregate gain.

The sensitivity case makes the intervention concrete. If the realized reduction is weaker than assumed, the projected error burden will be correspondingly higher. Human intervention therefore cannot be treated as optional polishing. Terminology-constrained MT with targeted human review is the selected response because it addresses the observed failure mechanism and creates a candidate for renewed adjudication. Any later release claim must pass the held-out rate and zero-critical-error gates and be tested, not assumed, through blinded domain-expert adequacy across held-out language–domain pairs.

My Five Release Rules for a Defensible 2026 Stack

A defensible 2026 release is a conjunction, not a ranking. Unedited MT is eligible only when the held-out, case-insensitive non-critical sentence-error rate clears its gate and every critical-error count remains zero. Among survivors, blinded domain-expert adequacy breaks the tie; a lower aggregate error rate cannot cancel a critical veto. I treat this ordering as the release policy, not as a claim that case-insensitive similarity establishes deployment safety.

I make the held-out language–domain pair, rather than a pooled development average, the certification unit. Before examining candidate outputs, freeze the normalization and casefolding procedure, adjudication guide, exclusions, and pairing policy. This prevents a favorable system comparison from being manufactured through unequal data or retrospective category changes. It also makes the proposed relationship falsifiable on pairs that did not influence model selection.

Casefolding creates a necessary edge case: it can conceal a required-casing violation. The aggregate comparison must therefore use casefolded text, while the critical audit retains the original surface form. Protected terminology, entities, numerals, negation, and mandatory case remain separate veto classes; moving an error into the non-critical bucket requires a documented adjudication decision, not a favorable score.

The terminology denominator is occurrence-level, not term-type-level and not segment-level. Repeated occurrences count as tested evidence, but the release record should preserve their language, domain, and protected-term identities. Otherwise, a small collection dominated by one frequent term can look stronger than the deployment-relevant terminology sample actually is.

For the adequacy tie-break, blind reviewers to candidate identity and randomize presentation order. Measure peak memory under the same hardware, numerical precision, batching policy, and input-length policy; otherwise, “lower memory” may merely describe a different serving configuration. The median is the decision statistic because a single highly fluent or highly defective segment should not dominate selection.

Encode the outcome in a release manifest containing the held-out pair, aggregate numerator and denominator, each critical count, terminology-occurrence count, exact uncertainty bound, median blinded adequacy, peak memory, certification date, and language–domain mix. According to the LoResLM 2025 overview, its workshop was held with COLING 2025 in Abu Dhabi, but the supplied abstract contains no 2026 experiment. These controls should therefore be presented as prospective, testable deployment policy—not as results validated by that workshop.

What to do next

Also worth reading: The secret to flawless machine translation accuracy: secret to flawless machine translation · How machine learning automates data extraction from hundreds of complex PDF layouts: How machine learning automates data · Why AI matters for precision in liturgical translation: Why AI matters for precision