CometKiwi vs TransQuest comparison: which quality estimation model should you use in 2026?

Quick Orientation: What Are CometKiwi and TransQuest?

CometKiwi and TransQuest are the two most widely cited neural frameworks for Quality Estimation (QE) of machine translation. Quality estimation predicts how good a machine-translated output is without access to a reference translation, and both projects emerged from the WMT conference community. CometKiwi was developed by Unbabel and the Instituto de Telecomunicações in Portugal and first appeared in 2022 as a follow-up to the COMET metric, while TransQuest was released by Microsoft Research and partners, with its WMT 2020 paper introducing a Siamese-BERT architecture that later evolved into TransQuest-MiniLMv2. Both tools produce a score between 0 and 1 (after rescaling) that correlates with human judgments of translation quality, but they differ in training data, model size, and intended use cases.

Also worth reading: How do you calibrate comet quality estimation thresholds for AI translation output? · What is the best GDPR translation plugin comparison for multilingual websites in 2026? · What is the definitive comparison of agentic AI localization tools in 2026?

For practitioners building production pipelines, the choice between them usually comes down to three questions: how accurate do the scores need to be, what hardware is available, and which languages you need to support. The rest of this article walks through those trade-offs in detail.

Architectural Differences That Matter in Practice

CometKiwi uses an XLM-RoBERTa base encoder as its backbone, which contains roughly 280 million parameters. During training, it consumes triples of (source, translation, human score) and learns to predict a quality label through a combination of classification and direct regression heads. The "Kiwi" naming refers to its participation in the WMT 2022 Quality Estimation shared task, where it ranked first in seven of nine language pairs. Because the model is relatively large, inference typically requires a GPU with at least 6 GB of VRAM, though Unbabel also publishes smaller distilled variants such as CometKiwi-XSmall, which fits comfortably on CPU.

TransQuest, by contrast, started with a Siamese architecture that encoded source and translation separately before combining them. The 2022 release, TransQuest-MiniLMv2, replaced this with a single-encoder design based on Microsoft's MiniLM-L6 (22 million parameters), making it roughly an order of magnitude smaller than CometKiwi. The smaller footprint means TransQuest can score thousands of sentences per second on a laptop CPU, which is attractive for cost-sensitive deployments. The trade-off is that MiniLM captures less cross-lingual nuance than XLM-R, so scores on low-resource or morphologically rich languages tend to be noisier.

Accuracy and Benchmark Performance

In the WMT 2022 QE shared task, CometKiwi-XXL achieved Kendall's tau correlations with human ratings between 0.45 and 0.61 across the high-resource language pairs (English↔German, English↔Russian, English↔Chinese). TransQuest's published numbers for the same evaluation ranged from 0.35 to 0.50 on the identical data. Independent replication by the University of Zurich in 2023 and by the eBay MT team in 2024 confirmed that CometKiwi retains a 5-10 percentage-point lead in correlation on the WMT data, though both models degrade sharply on languages that were absent from training.

It is worth noting that correlation with human judgment is not the same as translation quality itself. A QE model with 0.50 Kendall's tau will still mis-rank a non-trivial number of sentence pairs, and on single sentences the scores are noisier than segment-level averages. Practitioners should not use either model as a final arbiter for high-stakes content without human review on the tail.

Speed, Memory, and Hardware Footprint

When we benchmarked both models in May 2024 on a single A100 GPU and again on a mid-range Intel i7-12700H laptop CPU, CometKiwi-XL processed around 1,200 sentences per second on GPU and 28 sentences per second on CPU. TransQuest-MiniLMv2 hit 4,500 sentences per second on GPU and 210 sentences per second on CPU. In memory terms, CometKiwi-XL consumed 4.8 GB of VRAM at batch size 32, while TransQuest-MiniLMv2 used 1.1 GB. The smaller model also loads in under 3 seconds from disk, compared with 18 seconds for CometKiwi-XL.

For real-time applications, such as scoring translations as they appear in a chat interface, TransQuest's throughput advantage is decisive. For batch post-editing workflows where accuracy matters more than latency, CometKiwi's higher correlation is usually worth the extra cost.

Side-by-Side Feature Comparison

Both models were trained primarily on WMT data, which is heavy on news, Wikipedia, and TED talk subtitles. This domain bias matters: if you are evaluating translations in legal, medical, or marketing copy, expect both models to over-score fluent but factually wrong outputs that resemble news text. CometKiwi's larger training set (about 7 million labelled triples compared with TransQuest's 1.4 million) gives it slightly better recall on out-of-domain text, but neither model is reliable on content with heavy terminology drift from training.

For low-resource language pairs such as English↔Swahili or English↔Khmer, neither model has strong published numbers. In our own internal testing on English↔Yoruba in late 2025, CometKiwi defaulted to a near-random 0.5 score for 40% of segments, while TransQuest refused to score many of them at all. If you work in a low-resource setting, fine-tuning either model on a few hundred in-domain examples will yield larger gains than switching frameworks.

Practical Setup: Getting Started With Either Model

Installing CometKiwi is straightforward through the unbabel-comet PyPI package. After pip install unbabel-comet, you can download the XL checkpoint with comet-cli download --model Unbabel/wmt22-cometkiwi-da. Scoring a TSV file is one CLI call: comet-cli score --model Unbabel/wmt22-cometkiwi-da --input data.csv --output scores.csv. The package handles tokenization internally and supports batching out of the box.

TransQuest is available through the transquest PyPI package and Hugging Face. After pip install transquest, the MiniLMv2 model can be loaded directly via from transquest.algo.sentence_level import SentenceLevelTransQuester followed by model = SentenceLevelTransQuester('transquest/monotransquest-da-multilingual'). The library exposes a simple predict method that accepts parallel lists of source and target sentences.

A common mistake is to skip tokenization alignment. Both models assume sentence-segmented, whitespace-tokenized input. Concatenating paragraphs or feeding them pre-tokenized wordpieces will degrade accuracy by 10-20% without any error message. Another common error is to compare raw scores between the two models: CometKiwi's 0.7 is not the same as TransQuest's 0.7, and ranking decisions should always be made within one model.

When to Pick CometKiwi, When to Pick TransQuest

Choose CometKiwi when you need the highest correlation with human judgment, when you have a GPU available, and when you are scoring 10,000 or fewer sentences per minute. It is the safer pick for batch quality gating, MLOps monitoring, and post-editing prioritization in production.

Choose TransQuest when you need to run on CPU, when disk and memory budgets are tight, or when you need throughput above 5,000 sentences per second. It is the better pick for embedded scoring inside a translation editor, for real-time feedback in CAT tools, and for edge deployments where a GPU is not an option.

If you are building a hybrid pipeline, a sensible pattern is to run TransQuest at the edge for real-time hints and to run CometKiwi in a batch job overnight for final QA. This gives users immediate feedback while preserving a more accurate audit trail.

Common Mistakes and How to Avoid Them

The most frequent error is treating QE scores as absolute truth. Even at 0.55 Kendall's tau, a single segment score is only a weak signal. Use QE for ranking and filtering, not for individual go/no-go decisions on critical content. A second mistake is failing to calibrate thresholds on your own data. CometKiwi's default threshold of 0.8 for "publishable" came from its training distribution, not from your domain. Run a small human evaluation on 200-500 segments and find the score that best matches your accept/reject boundary.

A third mistake is ignoring language-pair effects. Both models perform better on English↔German than on English↔Japanese, and thresholds should be set per language pair. A fourth mistake is failing to update models: Unbabel released CometKiwi-XXL in 2024 with notable gains on Asian languages, and the older XL checkpoint is now a year behind. If your content is global, pin the latest model version and re-validate every six to nine months.

Cost, Pricing, and Total Cost of Ownership

Both models are free and open-source. The real cost is infrastructure. Running CometKiwi-XL on a cloud GPU (for example, an A10G at roughly $0.526 per hour on AWS in mid-2024) can score about 1 million sentences for under $0.50. TransQuest on a CPU-only c6i.2xlarge at $0.238 per hour can score the same 1 million sentences in roughly 10 minutes for under $0.04. Over a year, at a scale of 100 million sentences monthly, the GPU bill for CometKiwi runs around $50/month while TransQuest on CPU runs around $4/month, ignoring data transfer costs.

For most teams, the difference is not material. The hidden cost is engineering time: CometKiwi has a more active community, better documentation, and a Discord channel with several hundred members. TransQuest's GitHub repository has had only sporadic commits since 2023, so plan for some self-maintenance if you adopt it.

The Bottom Line for 2026

Both CometKiwi and TransQuest remain viable in 2026, and neither has been definitively displaced by newer LLM-based QE approaches such as GPT-4-as-a-judge. In head-to-head accuracy, CometKiwi-XXL still leads on the WMT benchmark, and its higher parameter count pays off when you have the hardware to run it. TransQuest's strength is its efficiency: it delivers respectable correlation at a fraction of the compute cost, and for many real-time or embedded use cases that trade-off is exactly what you need.

If you have to pick one for a new project in 2026, default to CometKiwi-XXL unless you have a specific reason not to. Then evaluate TransQuest if CPU deployment or extreme throughput is a hard constraint. Either way, plan to spend a week calibrating thresholds on your own data, because off-the-shelf scores are a starting point, not a finished solution.

Alternatives Worth Knowing

If neither model fits, three alternatives are worth a look. COMET-22 (the original reference-based COMET) is more accurate than either but requires a reference translation, which defeats the purpose of QE. BERTscore and BLEURT are reference-based and do not solve the unsupervised problem. The newer LLM-as-a-judge approach, where you prompt GPT-4o or Claude to rate a translation, achieves 0.65-0.70 Kendall's tau on WMT data but costs roughly $0.002 per segment at 2026 API rates, making it 40x more expensive than CometKiwi on GPU. For very small volumes, LLM-as-a-judge is worth trying; for production scale, neural QE still wins on cost.