Which evaluation metrics actually work for low-resource machine translation in 2026?

Why Low-Resource Translation Evaluation Is a Harder Problem Than It Looks

Low-resource translation evaluation is the discipline of judging machine translation quality when a language pair has little parallel data, limited monolingual corpora, and almost no professional reference translations. In practical terms, this covers any pair where the combined parallel training data falls below roughly 100,000 aligned sentence pairs, a threshold widely used since the early 2020s and still cited in 2026 benchmarking reports. Anything from Swahili-to-Icelandic to Tagalog-to-Basque falls into this category, and together these pairs cover roughly 70 percent of the world's languages as classified by cataloging language repositories. Standard metrics such as BLEU and chrF were tuned on high-resource European languages and assume dense n-gram coverage, so they routinely over-penalize valid alternative renderings in morphologically rich or low-resource languages. Add the fact that human reference sets for many African and South Asian languages contain fewer than 1,000 professionally reviewed sentences, and the problem becomes less about choosing a metric and more about building an evaluation system that survives unreliable references in the first place.

Also worth reading: COMET-QE vs LLM translation evaluation: which quality metric should you trust in 2026? · What is agent as judge translation and how does it improve upon traditional LLM-as-a-judge evaluation methods? · What are the definitive sovereign translation vendor evaluation criteria for organizations handling sensitive data in 2026?

The Limits of BLEU, chrF, and TER for Low-Resource Pairs

BLEU remains the most reported metric in academic papers, with a 2024 survey of 312 MT papers finding that 78 percent still cited it, but for low-resource work its Pearson correlation with human judgments often drops below 0.50. chrF, which operates on character n-grams, tends to hold up slightly better for languages with rich morphology such as Finnish or Yoruba, with correlations closer to 0.65 on those pairs. TER, the edit-distance-based metric, is useful when measuring post-editing effort but is brittle when the reference set contains multiple valid translations. The bigger issue is that these metrics require human references at all, and in low-resource contexts those references may be drafted by non-native speakers or produced through round-tripping, which inflates the apparent quality of fluent but semantically wrong outputs. Researchers publishing in the AAAI Conference on Artificial Intelligence have shown that sentence-level aggregation of lexical metrics correlates more strongly with human judgments than corpus-level aggregation, but even that finding depends on having at least a few hundred human-scored segments per language pair.

Learned Metrics: BERTScore, COMET, and Their Low-Resource Variants

Learned metrics replaced reference-based surface matching with semantic embedding comparison and moved the field forward between 2022 and 2025. BERTScore computes cosine similarity between contextual embeddings and works without exact surface matches, but it depends on the underlying language model having seen the target language during pre-training. For languages absent from those pre-training corpora, BERTScore degrades sharply and can be worse than BLEU. COMET, trained on direct human judgments from the WMT shared tasks, has become a de facto standard and its 2025 release (COMETKiwi) supports zero-shot scoring across roughly 200 languages. The practical catch is that COMETKiwi still uses an XLM-RoBERTa backbone, and its performance on truly low-resource languages such as Sinhala or Haitian Creole hovers around a Kendall correlation of 0.55, compared with 0.85 on English-German.

Using a frontier large language model as an evaluator has become the most active research area in 2025 and 2026. The TranslateGemma release from Google in early 2026 demonstrated translation ability across 55 languages, and its accompanying evaluation harness used Gemini-family judges to score outputs on a 1-5 scale. Independent replications of this approach report Kendall correlations between 0.60 and 0.75 against human judgments on high-resource pairs, but the same studies found a steep drop to 0.40 on pairs involving languages not present in the LLM's pre-training mix. The cost is also material: scoring 10,000 translation segments with a hosted LLM typically costs between $15 and $80 depending on the model and prompt length, compared with under $1 for BLEU and roughly $3 for COMETKiwi on local hardware. Latency is the other limiting factor, with LLM judging often running 50-100 times slower than embedding-based metrics. The honest read is that LLM-as-judge is the best single tool when you can absorb the cost and latency, but it should not be the only tool.

Practical Workflow for Evaluating Low-Resource Translations

The workflow that consistently produces trustworthy scores in low-resource settings combines four passes. First, run chrF or chrF++ against any available reference set, knowing that this establishes a noisy floor rather than a precise number. Second, run COMETKiwi in a zero-shot configuration to obtain a reference-free semantic score; this gives a second signal uncorrelated with surface overlap. Third, sample 200-300 segments and have a bilingual speaker rate them on a 1-5 adequacy and fluency scale, which yields roughly 2-3 hours of human work per language pair. Fourth, prompt an LLM judge with the same 200-300 segments using a rubric containing explicit definitions of adequacy and fluency, then compute the correlation between the LLM's scores and the human scores. If that correlation is below 0.50, the LLM judge is not reliable for that specific pair and should be down-weighted. This layered approach costs roughly $50-200 per language pair for the LLM step and 2-3 hours of expert time, but produces results you can actually act on.

Common Mistakes When Selecting a Metric

Three mistakes appear repeatedly in the evaluation literature. The first is reporting BLEU as the headline number for a low-resource language pair and treating a one-point improvement as meaningful; with 95 percent confidence intervals, BLEU differences below 2-3 points are usually noise on low-resource sets. The second is trusting an LLM-as-judge score without first calibrating it against a small human set, because LLM judges exhibit systematic biases toward longer outputs and more formal register, even when neither correlates with translation quality. The third is assuming that a metric which performs well on English-centric benchmarks will transfer to languages with different morphological typology; the 2026 Slator coverage of where AI translation struggles specifically calls out agglutinative and pro-drop languages as places where even strong metrics break.

When a Custom Metric Becomes Worth the Investment

Building a custom evaluation metric is justified when three conditions are met: the language pair matters enough to justify $5,000-20,000 in engineering effort, a human-rated validation set of at least 500 segments can be assembled, and the application has clear correctness criteria such as regulatory or medical content. Under those conditions, fine-tuning a 7-8 billion parameter multilingual model on the human ratings can produce a metric with Pearson correlations above 0.80, as documented in the 2025 Sarvam AI Indic ASR evaluation work for Indian languages. Outside those conditions, the off-the-shelf stack of chrF plus COMETKiwi plus a calibrated LLM judge covers roughly 90 percent of practical needs at a fraction of the cost.

Cost and Tooling Snapshot for 2026

Pricing and availability have shifted meaningfully since 2024. COMETKiwi remains free and open-weight, with inference possible on a single A100 GPU at roughly 50 segments per second. BERTScore is also free but requires you to host the backbone model. Hosted LLM judges cost between $0.001 and $0.01 per segment depending on the provider, with Anthropic's Claude and Google's Gemini offering the best documented results on low-resource pairs in 2026 benchmarks. Open alternatives such as Llama-3.1-70B and the TranslateGemma suite have closed much of the quality gap for languages they cover, but still lag hosted models on languages outside their pre-training mix. Human evaluation through platforms such as Prolific or Toloka runs at $0.10-0.50 per segment depending on language rarity and reviewer qualifications.

Final Recommendation

For most teams working on low-resource translation in 2026, the right baseline stack is chrF for surface reporting, COMETKiwi for reference-free semantic scoring, and a calibrated LLM judge validated against 200-300 human-rated segments for any go/no-go decision. Reserve custom metric development for high-stakes deployments where the $5,000-20,000 investment pays back through reduced human review cost or regulatory compliance. Treat single-number BLEU reports with skepticism, and always report confidence intervals rather than point estimates when the reference set contains fewer than 1,000 segments.