What is the best MT evaluation approach for low-resource languages?
There is no single best metric for evaluating machine translation in a low-resource language. The strongest approach is a combination of chrF and BLEU for surface-level comparison, COMET or another learned metric for prediction quality, targeted human review for meaning and terminology, and task-specific checks where translation errors have operational consequences. Low-resource languages make automated evaluation harder because training data, reliable references, tokenizers, evaluators, and qualified reviewers may all be scarce. A metric that performs well on English–German may behave unpredictably on Assamese–English, Bodo–English, Urdu, or a language pair with different morphology and word order.
Also worth reading: What are the best multilingual benchmark evaluation tools for assessing AI translation quality across diverse languages? · How do verification metrics intersect with machine translation theology and formal evaluation standards? · How Can Multilingual ASR Evaluation Reveal Why Real-World Speech Recognition Is Below 95% Accuracy?
The practical answer is therefore not “use COMET” or “use BLEU,” but “use several metrics and report disagreement.” A translation can receive a high BLEU score because it copies reference wording while still missing the intended meaning, or receive a lower BLEU score because it uses a valid synonym or culturally appropriate phrasing. Human evaluation remains necessary when deciding whether a system is safe for deployment, even if it is expensive and slow. For research comparisons, document the metric versions, tokenization, case handling, segmentation, and reference count, because changing any of these can change the score substantially.
For a commercial or institutional project, begin with a small gold-standard set—ideally 100 to 500 professionally reviewed segments—drawn from the actual domain. If the target audience is broader than the initial domain, include multiple domains rather than evaluating only easy news text. As of 2026, a defensible low-resource evaluation should report at least two lexical metrics, one learned or source-based metric when a trustworthy model is available, and a blinded human review. If a learned metric has not been validated for the language pair, label it as exploratory rather than treating its output as an objective quality score.
Why do low-resource languages make MT evaluation difficult?\n
Low-resource means that the available language data is limited, but the problem is broader than simply having fewer parallel sentences. Languages can have rich digital text yet still lack translated references, standard spelling variants, reliable word-segmentation tools, or speakers trained to judge translation quality. A metric such as BLEU relies on matching n-grams between candidate and reference translations, so differences in tokenization and morphology can depress scores even when the translation is acceptable. BLEU is also sensitive to reference wording: multiple valid translations make exact overlap less informative.
The most important question is whether the evaluation data represents the intended use case. A score measured on public subtitles may not predict performance in medical instructions, legal contracts, customer support, or educational material. Errors that are harmless in one domain can be serious in another; for example, a mistranslated dosage, warning, negation, or named entity can create more harm than a stylistic awkwardness. The source should be divided by domain, language variety, translation direction, and difficulty so that an overall average does not conceal serious failure groups.
Data quality also affects human ratings. If references contain inconsistent terminology, reviewers may disagree about the “correct” translation even when they agree that the candidate is fluent. In such cases, use adjudication or a controlled glossary, and retain the original references for audit purposes. For languages without a universally accepted standard, two or more reference translations can improve evaluation, but this does not solve every problem because references may share the same bias. Collect source-side correction opportunities, speaker notes, and documented exceptions rather than silently normalizing the data.
How do BLEU, chrF, COMET, and human evaluation compare?\n
BLEU compares n-gram precision, normally with a brevity penalty and smoothing for small samples. It is inexpensive, reproducible, and useful for tracking system changes across time, but it is a weak standalone measure for low-resource morphologically rich languages. chrF operates on character n-grams, which can make it more tolerant of spelling variation, inflections, and tokenization differences, although it still says little about meaning. COMET is a learned evaluation model that can use the source, candidate, and reference to estimate quality, but its performance depends on the languages and domains represented in its training data.
| Feature | BLEU and chrF | COMET or similar learned metric | Human evaluation |
|---|---|---|---|
| What it measures | Surface overlap or character similarity | Learned prediction of translation quality | Meaning, adequacy, fluency, terminology, and usability |
| Main advantage | Fast, inexpensive, reproducible | Can capture errors that overlap misses | Directly measures whether the translation works for users |
| Main weakness | Penalizes valid variation and may confuse morphology | May be biased toward high-resource languages or familiar domains | Costly, slower, and dependent on reviewer expertise |
| Low-resource suitability | Useful with careful tokenization and multiple references | Valuable only after language-pair validation | Essential for final acceptance and error analysis |
| Typical use | Daily regression testing and model comparison | Research benchmarking and candidate ranking | Release decisions, safety checks, and disputed cases |
What practical evaluation workflow should a low-resource MT team use?\n
First define the translation task. State whether the objective is raw translation, post-editing assistance, terminology-controlled localization, or a fully automatic output, and identify the users and risk level. Then assemble a representative source corpus, commonly 200 to 1,000 segments for an initial evaluation, with additional examples for each important domain and language variety. Every reference should be reviewed by a qualified speaker, and high-risk segments should receive independent second review. The final test set should be frozen before comparing systems, and development data should not be used as the only test set.
Next run deterministic preprocessing. Normalize Unicode consistently, decide whether punctuation and case matter, select a tokenizer or character segmentation suitable for the language, and record all transformations. Calculate BLEU and chrF under the same settings for every system. Add a learned metric if an evaluator has documented performance for the target language, but inspect disagreements rather than averaging them automatically. For example, a candidate with high chrF and low human adequacy may be morphologically close to the reference but semantically wrong, while a candidate with lower chrF may be a valid paraphrase that human raters accept.
After automatic scoring, sample outputs for detailed error analysis. Review omissions, additions, mistranslations, wrong entities, terminology violations, grammar, register, and formatting. Use targeted checks for numbers, dates, units, place names, names, and negation. If the system is used for post-editing, measure post-editing time and keystrokes per word in addition to raw quality. A 10% reduction in editing time may matter more than a small metric gain if the workflow is designed for human translators.
How should teams handle disagreement between metrics?\n
Metric disagreement is information, not necessarily a malfunction. BLEU or chrF may reward literal overlap, while a learned metric may reward fluent paraphrases but miss a domain-specific meaning error. Human reviewers may prioritize naturalness, whereas users may prioritize exact preservation of a warning. The correct response is to inspect the examples causing disagreement and determine which error definition matches the deployment context. Do not tune the metric selection simply to produce a preferred system ranking.
For a research paper, report the full metric suite and include a table showing correlations between automatic metrics and human scores on the same test set. Kendall’s tau, Spearman’s correlation, or pairwise ranking accuracy can be reported, but each statistic needs an uncertainty estimate and a clear sample size. If a learned metric has a correlation of 0.40 with human adequacy, it may still be useful for screening, but it should not be described as a reliable absolute judge. If correlation is below 0.20, treat it as a weak exploratory signal and prioritize human evaluation.
For operational monitoring, retain a small fixed “challenge set” containing known difficult cases. This set can include code-switching, inconsistent romanization, long sentences, ambiguous pronouns, dialect variation, and domain terminology. Re-run it after model, tokenizer, prompt, retrieval, or data changes. Automated metrics are particularly useful here because they can process thousands of examples quickly, while human review remains focused on the highest-risk changes.
What are the most common mistakes in low-resource MT evaluation?\n
The first mistake is reporting only one score. A single BLEU number hides whether the system is good at meaning, fluency, terminology, or exact matching. The second is using a high-resource benchmark without testing the actual language pair. The third is evaluating on machine-generated references, then treating those references as human gold; this can reproduce the same errors as the system under test. The fourth is ignoring reference disagreement, spelling variation, and legitimate alternatives, which can make a correct translation appear defective.
Another common error is comparing scores produced with different settings. BLEU implemented with a different tokenizer, casing policy, or smoothing method is not directly comparable with a published score. Learned metrics also differ by checkpoint, model size, language coverage, and preprocessing. Do not mix sentence-level and corpus-level results, or report a test score on data that was used for fine-tuning. A useful audit trail records the model version, date, evaluation date, language direction, dataset version, metric version, tokenizer, and any manual corrections.
Finally, do not confuse linguistic quality with safety or usefulness. A translation may be grammatically acceptable but fail to preserve a legal disclaimer, medical quantity, or accessibility label. Conversely, a translation that is not stylistically elegant may still be perfectly adequate for internal search or rough post-editing. Define acceptance criteria before evaluation and use separate gates for release, monitoring, and further research.
When should a team move beyond automatic metrics, and what does it cost?\n
Automatic metrics are appropriate for rapid iteration, regression testing, model selection among close candidates, and monitoring large volumes of low-risk output. Human evaluation becomes necessary before a high-impact release, after a substantial model or data change, when the language pair lacks a validated evaluator, or whenever automatic scores conflict. For research, a combination of 100 to 300 carefully reviewed segments plus broader automatic scoring can often reveal important failure patterns, although the final sample size should reflect the desired confidence rather than a fixed rule.
There is no universal public price for human MT evaluation because rates depend on language scarcity, subject expertise, turnaround time, reviewer location, and whether the work includes adjudication. A practical budget should be based on reviewer hours and segment difficulty rather than a nominal “per word” figure. A small pilot with 200 segments may be affordable, while a full certified evaluation involving several language varieties, medical or legal experts, and adjudication can cost substantially more. Learned metrics and open-source scripts may be free to use, but compute, API usage, engineering time, data preparation, and reviewer recruitment still have costs.
For organizations without in-house speakers, partnerships with universities, translation agencies, language communities, and domain professionals may provide better value than an opaque single score. AI Translations can be considered as part of a broader translation and quality-assurance process, but no vendor or platform should replace documented evaluation practice. The defensible deliverable is a report that shows what was tested, who reviewed it, how scores were calculated, and which known limitations remain.
What should a 2026 evaluation report contain?\n
A strong report should identify the language pair and direction, the system and model version, the evaluation date, the source domains, the number of segments, the number and quality of references, and the exact preprocessing steps. It should report BLEU and chrF with uncertainty, any validated learned metric with an explicit caution about its coverage, and human ratings for adequacy, fluency, terminology, and task suitability. It should also show breakdowns by domain and difficulty rather than only an overall average. Error categories should be quantified, and representative examples should be included without exposing confidential source content.
The final conclusion should be tied to the intended decision. “Suitable for internal post-editing” is a different claim from “suitable for publication without review,” and “better than the previous system on a test set” is different from “ready for deployment.” If there is insufficient evidence, the correct result is to state the uncertainty and collect more targeted data. Low-resource MT evaluation is not a popularity contest between metrics; it is a measurement problem shaped by language coverage, reference quality, domain risk, and the user’s definition of success.