What are the best on-device translation model benchmarks in 2026, and how do offline mobile translation models actually compare?

On-device translation model benchmarks measure how well machine translation models run entirely on a phone, tablet, or embedded device without sending text to a cloud server. As of August 2026, the field has matured rapidly: Tencent's HY-MT1.5 models (1.8B and 7B parameters) translate 33 languages fully offline from a package as small as 440 MB, Google released the TranslateGemma suite of open translation models built on Gemma, and Apple's on-device model for Apple Intelligence is evaluated against dedicated benchmarks alongside its cloud counterpart running on Apple silicon. This guide explains which benchmarks matter, how leading models score, what trade-offs you should expect, and how to evaluate an on-device translation model for your own use case.

What On-Device Translation Benchmarks Actually Measure

Also worth reading: How Are Multilingual ASR Benchmarks Evolving AI Translation? · How Do AI Translation Quality Benchmarks Measure Real-World Performance? · How Should You Design Translation Benchmarks for Reliable AI Evaluation in 2026?

A benchmark for on-device translation is not just a quality test. It combines at least four dimensions: translation quality, latency, memory footprint, and energy consumption. Translation quality is typically measured with COMET, chrF, or BLEU scores against human reference translations, with COMET being the current preferred metric because it correlates better with human judgment than older BLEU-based scoring. Latency benchmarks measure time-to-first-token and tokens per second on specific hardware — a number that means little unless the test device (a Snapdragon 8-series phone, an Exynos chip, a Raspberry Pi, or an Apple A/M-series SoC) is disclosed.

Memory footprint matters because mobile operating systems enforce hard limits on app memory. A 7B-parameter model quantized to 4-bit needs roughly 3.5–4 GB of RAM just for weights, which rules out most mid-range phones. Tencent's headline claim that its HY-MT1.5 model fits into 440 MB illustrates why quantization and architecture efficiency are now central benchmark categories rather than afterthoughts. Energy consumption, measured in joules per translated sentence or battery drain per hour of continuous use, separates models that are genuinely deployable from those that merely demo well.

Finally, coverage benchmarks count language pairs supported offline. Cloud systems can route any pair through a data center; an on-device model must carry every language pair in its weights, so the ratio of languages supported to megabytes consumed is one of the most honest efficiency metrics available.

The Current Leaders: Tencent HY-MT1.5, TranslateGemma, and Gemini Nano

Tencent's HY-MT1.5 family, released in 2026, currently defines the state of the art for compact on-device translation. The 1.8B variant was designed specifically for phones and edge devices, supporting 33 languages offline while occupying roughly 440 MB after aggressive quantization. Tencent reports that the 1.8B model approaches the quality of much larger general-purpose LLMs on standard WMT-style evaluation sets while running at interactive speeds on flagship mobile SoCs. The 7B sibling targets on-premise servers and high-end devices where more RAM is available, trading size for higher fidelity on low-resource pairs.

Google's TranslateGemma suite takes a different approach: open-weight Gemma-derived models fine-tuned specifically for translation, runnable through LiteRT (formerly TensorFlow Lite), Google's universal runtime for on-device AI. Because the weights are open, developers can quantize them further, distill them, or fine-tune them on domain-specific data — something closed APIs cannot offer. Google has also demonstrated specialized edge translation work beyond text, including sign-language recognition models designed to run locally on user devices via DeepMind research efforts.

Gemini Nano represents the third category: a general-purpose small language model embedded directly into the OS (Android via AICore, and similar patterns elsewhere). General-purpose small models can translate competently across many pairs but usually trail dedicated translation models like HY-MT1.5 on strict MT benchmarks by measurable margins, particularly on idiomatic phrasing and low-resource languages. The trade-off is flexibility — the same model handles summarization, rewriting, and translation.

Benchmark Comparison Table

This table simplifies deliberately. Real deployments should re-run quality evaluations on your own domain text, because public benchmark sets like WMT news test sets over-represent formal written language and under-represent chat, subtitles, and technical documentation.

How to Run Your Own Evaluation: Practical Steps

First, assemble a test set of 200–500 real sentences from your actual use case — support tickets, product listings, chat logs — ideally reviewed by native speakers. Public benchmarks alone will mislead you if your content differs from news-style prose. Second, pick two or three candidate models and run each through its intended runtime: LiteRT for TranslateGemma-class models, vendor SDKs for proprietary ones, and record tokens-per-second on your minimum-spec target device, not a flagship.

Third, score outputs with COMET (available via open-source tooling) plus a small human adequacy/fluency rating on a 1–5 scale from at least two native speakers per language. Human review of even 100 sentences catches failure modes that automatic metrics miss, such as dropped negations or untranslated brand names. Fourth, measure the operational numbers: cold-start load time, peak RSS memory, battery drain over 30 minutes of continuous translation, and behavior when the OS kills background processes. A model that scores 2 COMET points higher but takes 9 seconds to load and drains 15% of the battery per hour may still be the wrong choice for a travel app used in short bursts.

Finally, test degradation paths. What happens when the input language is misdetected? Does the model fail silently or produce garbage? Robustness to code-switching (mixed-language input) is a common blind spot in published benchmarks and a frequent real-world complaint.

Common Mistakes When Reading Benchmark Numbers

The most common mistake is comparing scores measured on different hardware or different quantization levels. A 7B model at FP16 and the same model at 4-bit can differ by several COMET points; a latency figure quoted on a Snapdragon 8 Elite tells you nothing about a mid-range Exynos device. Always ask: which checkpoint, which quantization, which chip, which runtime version?

The second mistake is trusting aggregate scores across all language pairs. Models often excel on high-resource pairs (English–Spanish, English–Chinese) while quietly degrading on low-resource ones. Ask for per-pair breakdowns, especially for the specific languages your users need. Third, beware of self-reported numbers without independent replication. Vendor claims — including Tencent's 33-language, 440 MB figure — are best treated as starting hypotheses until you reproduce them on your own test set. Fourth, do not equate general LLM leaderboard performance with translation ability. A model topping LMArena's human-preference rankings may still lose to a smaller dedicated MT model on strict translation metrics, because chat preference rewards fluency and style rather than fidelity to the source.

On-Device vs. Cloud Translation: When Each Wins

On-device translation wins when privacy is non-negotiable (medical, legal, enterprise messaging), when connectivity is unreliable (travel, aviation, field work), when latency must be near-zero, or when per-character API costs would scale badly. Running inference locally costs nothing per sentence beyond battery and silicon, whereas cloud APIs charge per character or per token and add 100–400 ms of network round-trip even under good conditions.

Cloud translation still wins on maximum quality for rare language pairs, on very long documents where context handling matters, and on teams that lack the engineering capacity to manage model deployment and updates. Hybrid architectures — on-device first, cloud fallback for hard cases or unsupported pairs — are increasingly the pragmatic default. Apple Intelligence exemplifies this pattern with its paired on-device and server-side models, both running on Apple silicon, routing requests based on complexity. Expect more vendors to adopt this split through 2027 as small-model quality keeps climbing.

Cost and Resource Considerations

Open-weight options such as TranslateGemma carry no licensing fee for the weights themselves but impose real engineering costs: integration time, QA across languages, storage in your app bundle (every 100 MB of model affects download conversion rates, particularly in markets with metered data), and ongoing maintenance as you update checkpoints. Quantization to 4-bit typically cuts memory by 60–75% relative to FP16 with modest quality loss — often under one COMET point — making it the default choice for shipping.

Proprietary on-device SDKs may involve revenue-share or platform requirements, while cloud APIs typically price from fractions of a cent per thousand characters up to enterprise contracts. For a consumer app translating, say, 50 sentences per active user per day, local inference eliminates a recurring bill that grows linearly with your user base — often the decisive financial argument once daily active users pass the tens of thousands.

When to Act and What to Watch Next

If you ship translation features today, the practical move is to prototype with HY-MT1.5-class dedicated models or TranslateGemma on LiteRT, validate against your own test set, and reserve cloud calls for fallback. If you are planning for late 2026 and 2027, watch three developments: continued shrinkage of dedicated MT models below 1B parameters with retained quality, better on-device support for speech-to-speech pipelines (where sign-language and voice translation research is heading), and standardized edge-AI benchmark suites that report quality, latency, and energy together rather than separately.

One caution: benchmark leadership in this space rotates quickly. A model that leads in June can be overtaken within two quarters, as the rapid succession of releases throughout 2026 demonstrates. Build your evaluation pipeline so that swapping models is a configuration change, not a rewrite. That discipline — not loyalty to any single model — is what keeps translation quality high as the field moves.