HTER vs BLEU for translation quality: which metric should you actually use?

What HTER and BLEU Actually Measure

Human-targeted Translation Error Rate (HTER) and Bilingual Evaluation Understudy (BLEU) answer fundamentally different questions about a translation, which is why the HTER vs BLEU debate keeps surfacing in localization teams. BLEU is an automatic, n-gram-overlap metric that compares a machine translation output against one or more human reference translations. It counts matching 1- to 4-gram sequences, applies a brevity penalty, and produces a score between 0 and 1 (often shown as 0–100). HTER, by contrast, is a human-in-the-loop metric: an editor post-edits a machine-translated segment, and HTER is the ratio of edit operations required to bring that output up to publishable quality, expressed as a percentage of the source tokens.

Also worth reading: AI translation cost comparison 2026: how much do GPT-5.6, Google Gemini, and dedicated tools actually charge per word or per minute? · what is AI translation and how does it actually work in practice? · How do I build a professional MTPE quality dashboard setup for enterprise translation?

The two metrics were designed for different stages of the workflow. BLEU was introduced by Papineni et al. in 2002 as a fast, reproducible proxy for adequacy and fluency, and it became the default benchmark for academic MT papers for two decades. HTER was formalized by Snover et al. in 2006 specifically to estimate the post-editing effort a human would need to invest, which is closer to what a paying client actually cares about. In a 2024 Frontiers in Computer Science study on MT error types and post-editing effort, researchers found that certain error categories (mistranslated numbers, named entities, and negation) drive HTER up far more than they depress BLEU, because BLEU rewards any plausible n-gram overlap even when the meaning is wrong.

Why BLEU Alone Is No Longer Enough

BLEU has well-documented weaknesses that became more visible as neural and, later, large language model (LLM) translation systems matured. It is insensitive to meaning preservation, rewards lexical overlap over semantic equivalence, and can be gamed by systems that produce fluent but inaccurate text. A 2023 IBM Research retrospective on multilingual NLP noted that BLEU's correlation with human judgment varies wildly by language pair, hovering around 0.6 for high-resource English-German but dropping below 0.3 for low-resource African and South Asian languages. The same paper observed that since the introduction of Transformer-based systems in 2017, BLEU gains of 1–2 points have often failed to translate into perceptible quality improvements for end users.

LLM-based translation has made the problem sharper. In a 2026 AWS evaluation guide for running LLMs on machine translation tasks, the authors recommend supplementing BLEU with semantic metrics such as COMET, BLEURT, and chrF, plus task-based evaluation, because BLEU underweights the very fluency that LLMs excel at producing. A MarkTechPost report on Gradium's 2025 speech translation launch showed that Gradium's s2s-translate beat gpt-realtime-translate on accuracy and latency, yet the BLEU gap was only 1.4 points, while human-rated accuracy differed by 11 percentage points. That gap is exactly the territory where HTER or direct human evaluation becomes decisive.

Where HTER Shines and Where It Breaks Down

HTER's strength is its direct connection to cost. If your average HTER on a domain is 15%, a post-editor will need to touch roughly 15% of the source tokens, and you can price the job accordingly. This makes HTER the metric of choice for translation buyers running post-edited MT (PEMT) pipelines at scale, and it is the metric most often cited in vendor scorecards. The Frontiers in Computer Science study quantified this link: each 10-point increase in HTER correlated with an additional 4.2 minutes of post-editing time per 1,000 source words across the 18 professional translators in the sample.

The downsides are equally concrete. HTER requires a trained post-editor, costs roughly $0.04–$0.12 per source word in 2025–2026 market rates, and is slow to produce. It is also sensitive to editor style: a heavy-handed editor who rewrites for house style will report higher HTER than a minimal-edit editor on the same output. BLEU, by contrast, is free, instant, and perfectly reproducible, which is why it still anchors academic benchmarks and A/B testing of model variants. The honest answer to HTER vs BLEU is that they measure different things, and most mature programs use both.

A Direct Comparison

Start by defining the decision the metric will drive. If you are selecting between two MT engines for a new language pair and need a reproducible signal across thousands of sentences, run BLEU (or better, chrF and COMET) on a held-out test set of at least 1,000 segments. If you are renegotiating a vendor SLA or deciding whether a domain is ready for raw MT without post-editing, run HTER on a 2,000–3,000-word pilot with two independent editors and average their scores. The Frontiers study recommends a minimum of 1,500 words per editor per domain for HTER to stabilize within ±2 points.

For ongoing monitoring, a tiered approach works well in practice. Use BLEU weekly as a smoke test to catch catastrophic regressions (a sudden drop of more than 3 BLEU points usually signals a broken model or pipeline). Use HTER monthly on a rotating sample of live content to track real editing cost. Supplement both with targeted human review on high-risk content: legal clauses, medical instructions, and culturally sensitive material such as the children's stories discussed in a 2024 Nature paper on neuro-symbolic English-to-Tamil translation, where emotional paraphrasing can shift BLEU without changing acceptability.

Common Mistakes When Using These Metrics

The most frequent error is treating BLEU as a quality verdict rather than a proxy. Teams celebrate a 0.8-point BLEU improvement on a new model while ignoring that the same model mistranslates 4% of named entities, a defect HTER would surface immediately. The inverse mistake is equally common: rejecting a perfectly usable MT system because HTER on a 500-word pilot came in at 22%, when the underlying cause was a single editor's house-style preference rather than systemic error.

A subtler pitfall is comparing BLEU across language pairs. A BLEU of 35 means very different things for English-German (mediocre) and English-Finnish (strong), because reference length and morphology shift the baseline. The 2026 AWS guide explicitly warns against cross-pair BLEU comparisons and recommends per-pair baselines instead. Finally, do not average HTER across editors without checking inter-annotator agreement; if two editors on the same content differ by more than 8 HTER points, your editor guidelines need work before the metric can be trusted.

When BLEU and HTER Are Not Enough

For high-stakes or creative content, neither metric tells the whole story. Literary translation, marketing copy, and audiovisual subtitling demand human evaluation against rubrics that capture voice, register, and cultural fit. A 2025 Nature paper on AI-enhanced audiovisual translation for Chinese film and television found that BLEU and HTER together explained only 58% of variance in audience-rated naturalness, because viewers penalize awkward lip-sync and culturally flat phrasing that neither metric detects. Similarly, the Gonzo ML multi-agent collaboration paper on ultra-long literary translation argued that chapter-level coherence and character voice consistency require human or LLM-as-judge evaluation rather than sentence-level metrics.

For sign language translation, where timing and visual grammar matter, a 2025 Nature paper on the ADAT time-series-aware transformer showed that BLEU on gloss sequences correlated poorly with comprehension scores from Deaf evaluators. In these domains, the practical recommendation is to use BLEU and HTER as triage filters, then route anything that passes to a domain-specific human or rubric-based review.

Cost, Pricing, and ROI in 2026

Running BLEU is effectively free: open-source toolkits such as SacreBLEU produce scores in seconds on a laptop. HTER costs scale with editor time. At 2025–2026 market rates of $0.04–$0.12 per source word for post-editing, a 5,000-word HTER pilot costs $200–$600 per editor, so a two-editor pilot lands at $400–$1,200. For a localization program processing 2 million words a month, a 1-point HTER reduction translates to roughly $800–$2,400 in monthly editing savings, which is why even modest metric-driven improvements pay back the evaluation cost within weeks.

LLM-as-judge evaluation sits in between: API costs of a few cents per segment, near-instant turnaround, and correlation with human judgment in the 0.7–0.85 range for high-resource pairs according to the AWS 2026 guide. For teams that cannot afford full HTER studies, an LLM judge scored against a small human calibration set is a defensible middle ground, provided the calibration is refreshed quarterly.

The Bottom Line

HTER vs BLEU is not a contest; it is a division of labor. Use BLEU (ideally alongside chrF and COMET) for fast, cheap, reproducible model selection and regression detection. Use HTER when you need to know what a translation will cost to finish or whether raw MT is safe to ship. Use human rubric evaluation or LLM judges for creative, high-stakes, or culturally sensitive content where neither automatic metric is reliable. The programs that get translation quality right in 2026 are the ones that treat metrics as a portfolio, not a single number to maximize.