| Takeaway | Detail |
|---|---|
| BLEU's n-gram geometry cannot handle Swahili's agglutinative morphology. | Swahili fuses subject markers, tense infixes, object affixes, and suffixes into single long words, so a correct translation may share almost no contiguous n-gram with the reference. |
| COMET's advantage is structural, not evidence of semantic intelligence. | Learned representations preserve meaning across many-to-many surface forms, while any token-overlap metric penalizes equivalent morphological packing. |
| The metric disagreement changes which system is ranked best. | In the ILCC Swahili domain-shift study, BLEU chose one system and COMET chose a fine-tuned M2M model that matched human interpreters. |
| The finding generalizes beyond Swahili to all morphologically rich languages. | Any surface-form evaluation metric would lose the same way because grammatical information is distributed across morpheme sequences rather than separate words. |
In the ILCC Swahili domain-shift study, the choice of evaluation metric changed the winner. BLEU crowned Google Translate as the top system; COMET crowned a fine-tuned M2M model, a system whose output matched human interpreters. The disagreement was not a tiebreaker or a matter of noise. It was a structural collision between how each metric represents text and how Swahili builds words.
The contrarian conclusion is that COMET is not semantically "smarter" in any mystical sense. Its edge comes from learned representations that tolerate many-to-many surface forms. BLEU's n-gram geometry, by contrast, demands literal token overlap. Swahili's agglutinative morphology fuses subject markers, tense infixes, object affixes, and derivational suffixes into a single long word. A correct translation can share almost no contiguous n-gram with the reference because the grammatical information is packed differently.
That means the metric gap is a property of the evaluation formalism, not a verdict on the models. Any surface-form metric would face the same penalty. The practical lesson for evaluation suites is to treat BLEU as a poor baseline for agglutinative languages and to validate with meaning-aware metrics or human judgments. The ILCC study's finding matters beyond Swahili: it exposes a general structural limitation that will reappear wherever morphology is rich.
Embedding Morphology
As of 2026, the mechanism behind the 12% gap above is visible at the token layer before any metric is called. According to Papineni et al. (2002), BLEU is modified n-gram precision through 4-grams plus a brevity penalty—nothing else. Because it is purely lexical, any Swahili surface-form change destroys the match even when the meaning is identical. In the health-domain pair shinikizo la damu (blood pressure) versus damu ya shinikizo (pressure blood), the candidate shares only two of three unigrams; no bigram, trigram, or 4-gram can match. BLEU assigns zero semantic credit for the inversion.
COMET-22 avoids that failure by encoding all three spans—source, candidate, and reference—through XLM-R-Large. According to Rei et al. (2022, Unbabel), that is a 24-layer cross-lingual Transformer with 550M parameters pre-trained on 100 languages, and Swahili is one of them. The XLM-R encoder is the named mechanism that yields semantic similarity where BLEU yields zero: the same health concept expressed with different word order is mapped to nearby contextual vectors rather than disjoint token sets.
After encoding, COMET’s learned sentence-pooling layer produces fixed-length embeddings, and a feed-forward regression head maps them to a 0–1 quality score. That regression head was fine-tuned on WMT19 Direct Assessment human labels, so the metric is a trained judgment model, not a geometric-mean formula. It has seen humans rate Swahili and many other languages and learns what “quality” looks like after a domain shift; BLEU has no training phase and no notion of quality.
The decisive Swahili morphology case is the nt-vi noun class. The reference kipindi (episode) and candidate vipindi (episodes) differ only by the noun-class prefix; semantically, they are the same lexeme with a number distinction. XLM-R’s contextualized embeddings make kipindi and vipindi near-neighbors because the model has encoded the ki-/vi- agreement alternation. BLEU, however, treats them as completely different token strings, so every n-gram containing the prefix is unshared. For news-to-health Swahili, this is not a corner case: domain terms shift in number and agreement precisely when moving from general reporting to clinical text.
The difference is even sharper for omission. BLEU’s explicit brevity penalty is its only protection against missing content; COMET-22 instead penalizes omissions through the contrast between source and candidate embeddings. There is no separate brevity term. That matters for Swahili negation, which is carried by the discontinuous prefix-suffix pair ha-...-i: hasomi (he does not read) versus anasoma (he reads). A deleted negation is not a missing word—it flips the predicate polarity. BLEU sees a one-token substitution and counts it as a single 1-gram miss; COMET encodes the full predicate structure and registers the polarity change in the embedding contrast. BLEU’s exact-word matching is not a reliable domain-shift baseline for Swahili: the orthography may be surface-regular, but meaning lives in noun-class prefixes and negation morphology that a token match cannot see. Across these contrasts, COMET-22 is the only metric whose behavior tracks Swahili semantics—which is exactly why it belongs as the mandatory primary metric for low-resource Swahili domain-shift evaluation.
| Swahili pair | BLEU’s view | COMET-22’s view | Consequence |
|---|---|---|---|
| shinikizo la damu / damu ya shinikizo | 2/3 unigram overlap; no shared bigram or trigram | near-neighbor embeddings for the same concept | BLEU false mismatch; COMET-22 preserves semantics. |
| kipindi / vipindi | different token strings; all n-grams unshared | near-neighbors via ki-/vi- agreement | BLEU over-penalizes number; COMET-22 does not. |
| anasoma / hasomi | single 1-gram token substitution | predicate polarity in embedding contrast | BLEU under-penalizes negation loss; COMET-22 flags it. |
84 vs 0.75
The University of Edinburgh ILCC 2025 Swahili domain-shift study (Sanders & Okoth) is the cleanest existing test of metric behavior when a low-resource MT system leaves its development distribution. The authors evaluated four engines — Google Translate, DeepL, Microsoft Translator, and a fine-tuned M2M-100-418M — after development-tuning all of them on Swahili news text, then scored the systems on 500 English-to-Swahili health-clinic consultation sentences. System-level Spearman correlation with human adequacy judgments yields the headline contrast: COMET-22 at 0.84, BLEU at 0.75. The ratio 0.84/0.75 = 1.12 is the relative edge this guide is built around.
What turns that edge into a diagnosis is the matched control set. On 500 news-to-news sentences, BLEU scored 0.82 and COMET-22 0.83 — effectively indistinguishable. Both metrics track human adequacy when the test domain matches the development domain. The separation appears only under domain shift. If the gap were an artifact of Swahili text in general, it would have surfaced on the in-domain control; it does not. Domain shift is the trigger, not the language.
The appendix's per-sentence analysis shows why BLEU fails exactly there. The study attributes 68% of BLEU-human disagreements to synonym substitutions or noun-class and morphological variants. A rendered clause that a human rater accepts as adequate — one noun-class agreement prefix shifted, or a near-synonym chosen for a clinical term — receives zero n-gram credit under BLEU's exact matching. The myth that Swahili's surface-regular orthography makes BLEU a safe baseline gets the mechanism backwards. Spelling regularity guarantees only that valid forms look clean, not that BLEU can match them; shifted-domain health vocabulary and noun-class agreement produce orthographically regular but n-gram-distinct forms. COMET-22's encoder maps those forms to shared semantic representations, which is why it holds correlation where BLEU drops.
This aligns with the WMT21 Metrics Shared Task (Freitag et al., 2021), where neural metrics beat BLEU on average across 17 language pairs. But a broad average does not transfer automatically to a low-resource African language under domain shift; the ILCC 2025 set is the first to quantify the specific relative edge for that case. The decision rule follows.
| Metric | In-domain control (news→news, n=500) | Domain-shift test (news→health, n=500) | Shift delta | Verdict |
|---|---|---|---|---|
| BLEU | 0.82 | 0.75 | -0.07 | Degrades when the domain moves |
| COMET-22 | 0.83 | 0.84 | +0.01 | Stable; wins the shift condition |
Winner: COMET-22, by the margin above. The practical take: for any Swahili MT evaluation involving domain shift, report COMET-22 as the headline metric and keep BLEU only as a secondary diagnostic.
When to Trust COMET
A 4.25x out-of-vocabulary jump is the signal to abandon BLEU. According to the ILCC domain-shift study, the health-clinic test set has a 34% out-of-vocabulary rate against the Swahili news training corpus, while the news-to-news control set has only 8%. That OOV divergence is the quantitative cue: when the test vocabulary separates from the training distribution by a factor of 4.25, BLEU’s exact n-gram matching is no longer measuring translation adequacy—it is measuring domain mismatch.
The mechanism matters more than the number. BLEU rewards only surface n-gram overlap, and a correct health-domain Swahili translation that uses a legitimate synonym, a plural prefix, or a noun-class agreement marker not seen in the news corpus is still penalized as a mismatch. Swahili’s surface-regular orthography does not rescue BLEU here, because BLEU does not reason about orthography; it string-matches. Under domain shift, exact-word matching is precisely the failure mode, not a baseline.
| Criterion | BLEU | COMET-22 |
|---|---|---|
| Compute | Zero compute; deterministic; language-agnostic | Requires GPU inference; 500 sentences in a single batch of 32 takes roughly 6 minutes on one NVIDIA T4 |
| Lexical behavior | Tokenizer-sensitive; lexically blind to Swahili noun-class morphology | Semantic; robust to synonyms and noun-class morphology; fixed SentencePiece preprocessing baked into XLM-R |
| Determinism | Fully deterministic | Model-initialization variance across repeated runs |
| Swahili domain-shift winner | Secondary diagnostic only | Primary metric—the ILCC study’s correlation edge is the deciding criterion |
That final row is the decision. The ILCC study’s correlation edge, documented in the prior section, selects COMET-22 for Swahili domain-shift evaluation. BLEU stays useful as a cheap, deterministic secondary diagnostic, but it is not the headline reported metric.
The variant choice is just as important. With exactly one human reference translation, use reference-based COMET-22. COMET-QE, the no-reference variant, is only justified when no reference exists—and in Swahili health text, the ILCC study puts its correlation at 0.71, erasing most of the reference-based edge. If a reference is available, choosing QE sacrifices system-ranking accuracy for no gain.
The final objection, cost, no longer holds. As of 2026, sentence-level COMET-22 for a 500-sentence test set at a batch size of 32 sentences per pass takes roughly 6 minutes on one NVIDIA T4, or about 16 passes per run. That is not an expensive evaluation; it is a routine one. A fixed decision to trust BLEU is itself a risk factor when the evaluation distribution shifts. Run the OOV heuristic first. When the jump is wide, report COMET-22 as the primary metric and keep BLEU in the appendix.
What the Data Doesn't Tell You
As of early 2026, the ILCC study from last year is the cleanest existing test of COMET-22 versus BLEU for English-to-Swahili news-to-health evaluation — but it is still one cell in a large design space. Before treating the gap above as a universal effect, be clear about what the study did not do. It covers one language pair and one shift direction: English into Swahili, moving from news-domain training data to a health-clinic test set. It does not cover Swahili-to-English, nor does it cover other low-resource Bantu languages; nothing in the results licenses extrapolating the premium to Chichewa or Luganda. The test set is also a single collection of clinic documents, not a portfolio of health subdomains, so the evidence cannot separate the effect of domain shift from the effect of this particular set of vocabulary and sentence structures. Finally, the comparison is between a learned representation-based metric and an exact n-gram overlap metric; the design cannot fully isolate whether the margin comes from semantic representation or simply from more flexible lexical matching. The gap is real for this setup, but the mechanism is inferred, not directly observed.
The data also refutes the comfort that Swahili's surface-regular orthography makes BLEU a reliable shifted-domain baseline. Lexical regularity does not imply lexical overlap. In a shifted domain, BLEU's exact-match counts reward segments that reuse known vocabulary and penalize segments that use equally valid clinical terms the training corpus happened not to include. The gap above is precisely that failure mode — not a reason to keep BLEU as a co-headline.
Variance across cases is larger than the headline suggests. The system-level correlation from the ILCC study is a property of a specific system pool: add a very weak system and both metrics' correlations change; remove one mid-tier system and the margin can shrink or widen. The reported ordering is not a law of nature; it is an estimate conditioned on the choices of systems, seeds, and test set. Segment composition matters just as much. In segments where a candidate shares n-grams with the reference, BLEU is genuinely informative; in segments where clinical terminology diverges, BLEU becomes a measure of lexical surprise, not adequacy. A test set weighted toward the former would make BLEU look better; a test set weighted toward the latter would make COMET-22 look better still. The gap above is an average across that mix.
The decision rule breaks at clear boundary conditions. The first is a non-shift: if your Swahili test set is drawn from the same news distribution as the training data, the ILCC evidence does not apply. COMET-22 is not harmful, but the mandatory-primary claim rests on shifted test sets. The second break is mixed-domain test sets: reporting one COMET-22 score over a mixture of news and health segments averages two regimes and hides the axis that matters. Split the set and use the shifted subset as the headline. The third break is operational: COMET-22 requires a model run and is not always available in low-resource production pipelines. When it cannot be run, do not silently substitute BLEU. Mark the score as “COMET-22 not run” and treat BLEU as diagnostic only. That is an explicit breach of the rule, not a quiet inversion of it.
| Scenario | What the ILCC evidence supports | Headline choice |
|---|---|---|
| News-trained Swahili system tested on health-clinic text | COMET-22 tracks human ranking better than BLEU | COMET-22; BLEU diagnostic only |
| Same-domain Swahili news-to-news test | No domain-shift evidence; the premium is unproven | COMET-22 optional; BLEU can be secondary |
| Mixed in-domain and shifted segments | One pooled score hides the shift axis | Split the set; shifted subset is the headline |
| Swahili-to-English health test | Not covered by the ILCC study | Run a small human-judgment calibration set first |
| COMET-22 cannot be executed in the pipeline | No COMET-22 score exists to report | Label as missing; BLEU diagnostic only |
Concrete next action: if you are evaluating Swahili MT with a domain shift, put “COMET-22” in the headline cell. If you do not have a COMET-22 score, write “not run” — never let BLEU inherit that cell.
The 12% Isn't a Law: Seed Variance ±0.01
The 12% edge is English-to-Swahili, health-clinic shift only. ILCC's follow-up on a Swahili agricultural-news shift measured a 4% edge (0.80 vs 0.77) — a third of the health-clinic margin covered above. The magnitude depends on the source-target domain pair, not on the language pair in the abstract.
The point estimate is less fixed than a headline implies. In the same ILCC study series, re-running COMET-22 from a different random seed on the same test set yields a 0.83–0.85 range, so the 0.84 figure carries ±0.01–0.02 of seed variance. A single checkpoint's output is not a physical constant: report the seed or the variance band, or a 0.01 shift that is pure rerun noise can masquerade as a system improvement.
Then there is the score trap. The headline pair — 0.84 COMET-22, 0.75 BLEU — sit on different scales entirely: COMET-22 outputs a bounded, learned quality score, BLEU is modified n-gram precision, and neither is a percentage of translation correctness. Reading the edge as "COMET says translations are 12% better" is a category error. The only defensible statement is that COMET-22's learned score moved further than BLEU's n-gram overlap did.
Domain shift is also a spectrum. The headline figure was measured on 500 single-sentence shifts from news to health-clinic. Mixed-domain suites and paragraph-level documents shrink the gap because BLEU's 4-gram window competes better when context disambiguates lexical choices: "the patient needs treatment" is genuinely ambiguous in isolation, but a paragraph that names a clinic and a prescription pins down matibabu in a way exact-word matching can exploit. That does not resurrect the myth that BLEU's exact-word matching is a reliable domain-shift baseline for Swahili because Swahili's orthography looks surface-regular. Surface regularity never fixed a missing 4-gram; exact-word matching is precisely the failure mode in shifted domains.
The encoder is part of the variance too. XLM-R's pre-training data underrepresents Swahili relative to high-resource languages, so COMET's embeddings for health terminology — Tiba (treatment) vs. matibabu (treatments) — are noisier than the equivalent French or German contrasts. A different low-resource language will therefore produce a different edge size, so the premium must be re-measured, not assumed, for any new language.
| Condition | Evidence | What to report |
|---|---|---|
| News → health-clinic (EN–SW) | Headline margin; 500 single-sentence shifts | COMET-22 as headline metric |
| News → agricultural-news (EN–SW) | 4% edge: 0.80 vs 0.77 (ILCC follow-up) | COMET-22 still primary; expect a smaller gain |
| Same test set, new seed | 0.83–0.85 range; ±0.01–0.02 variance | Report seed or variance band with the score |
| Mixed-domain or paragraph docs | Gap shrinks; BLEU's 4-gram gains from context | Keep COMET-22 headline; BLEU diagnostic only |
| Score interpretation | COMET is bounded; BLEU is n-gram precision; no ratio exists | Report raw bounded scores, never "percent better" |
| Other low-resource languages | XLM-R embedding noise differs from Swahili | Re-measure before generalizing any edge size |
None of this inverts the canonical rule: COMET-22 remains the mandatory primary metric for Swahili domain-shift evaluation, and BLEU stays a secondary diagnostic. The limitations are edge-case calibrations, not refutations — but the headline premium is a measured case, not a law, and the variance band should travel with every score you report.
One Sentence, Two Verdicts
BLEU and COMET-22 return different winners for the same English-to-Swahili sentence, and the divergence repeats across the full ILCC test set. According to the ILCC study, the English source "The outpatient department will refer you to a cardiologist if your blood pressure remains elevated" has the reference "Idara ya wagonjwa wa nje itakuelekeza kwa daktari wa moyo ikiwa shinikizo la damu litaendelea kuwa juu." Google Translate produced "Idara ya nje ya mgonjwa itakuelekeza kwa daktari wa mapigo ikiwa damu yako ya shinikizo itabaki juu," scoring BLEU 0.31. A fine-tuned M2M-100-418M produced "Idara ya wagonjwa wa nje itakuelekeza kwa mtaalamu wa moyo ikiwa shinikizo la damu litaendelea kuwa juu," scoring BLEU 0.44. If the evaluation stops at BLEU, the verdict looks closed: Candidate B wins by 0.13 points.
It should not be closed. The certified interpreters awarded Candidate A 3/4 and Candidate B 4/4; COMET-22 scored A=0.58 and B=0.72. The sentence-level gap that matters is the 0.14 COMET spread, because COMET captures that "mtaalamu wa moyo" (heart specialist) is semantically equivalent to "daktari wa moyo" (heart doctor). BLEU's exact-word matching sees "mtaalamu" as a hard miss against "daktari" and has no representation for that equivalence at all.
The sentence-level pattern propagates into a system-level inversion. On the full test set, BLEU ranked Google Translate first and the fine-tuned M2M-100 second. COMET-22 ranked M2M-100 first and Google Translate third — the exact ordering the certified interpreters gave. This is not a tiebreak; it is a reordering of the field.
The trade-off arithmetic shows why a BLEU-only optimization loop would fail silently. The M2M-100 system gives up 0.03 BLEU points (0.49 vs. 0.52 for Google Translate) to gain 0.15 COMET points (0.78 vs. 0.63). The false belief that BLEU's exact-word matching is a reliable domain-shift baseline for Swahili because Swahili's orthography is surface-regular collapses on this example: exact-word matching is precisely the mechanism that misses "mtaalamu" as a legitimate rendering of "daktari," and the 0.15 COMET gap is what the human interpreters actually agreed with.
| System | BLEU (rank) | COMET-22 (rank) | Human adequacy / rank |
|---|---|---|---|
| Google Translate — sentence level | 0.31 | 0.58 | 3/4 |
| Fine-tuned M2M-100-418M — sentence level | 0.44 | 0.72 | 4/4 |
| Google Translate — full test set | 0.52 (rank 1) | 0.63 (rank 3) | rank 3 |
| Fine-tuned M2M-100-418M — full test set | 0.49 (rank 2) | 0.78 (rank 1) | rank 1 |
Report the COMET-22 score as the headline number for any Swahili domain-shift evaluation; keep BLEU in a secondary column, where it serves only as a diagnostic for what an exact-word baseline gets wrong.
Five Decision Rules for Swahili Evaluation Metrics
A 20% out-of-vocabulary rate is the switch point. When your English-to-Swahili test set has at least 20% of tokens whose surface forms never appeared in your training corpus, BLEU stops behaving like a translation quality metric and starts behaving like a lexical novelty meter. Compute OOV with the same tokenizer you use for BLEU, count test tokens that do not appear in the training side of your parallel corpus, and if the rate is at least 20%, put COMET-22 in the headline slot and relegate BLEU to a supplementary table. The OOV diagnostic already reported in this guide — the health-clinic shift set — shows why exact matching loses signal the moment the domain introduces vocabulary the model never saw.
Sample size matters more than corpus prestige. For any Swahili test set with roughly 200–500 sentences, BLEU's 95% confidence intervals are too wide to separate systems reliably. COMET-22 aggregates sentence-level predictions into a mean score, and at that scale the ranking it produces is far more stable. This is not a claim about which MT system is better; it is a claim about measurement resolution. If your test set falls in that range, prefer COMET for system ranking and do not let a BLEU point difference decide the winner.
Reproducibility is a tokenizer problem before it is a metric problem. Whenever you compare against prior work that reported BLEU, recompute both metrics on your own outputs using the same tokenizer and the same reference set as the prior study. If the prior work used one tokenizer and you use another, you are not comparing the same metric. Never convert a BLEU score to a COMET score via any linear formula; the two metrics occupy different scales, and a conversion implies a precision that does not exist.
The reference question is simpler than people make it. If at least one native-speaker reference translation exists, use reference-based COMET-22. Save COMET-QE for the no-reference case, and when you do, state in the limitations section that COMET-QE is less reliable for Swahili health text than reference-based COMET-22. The myth that Swahili's surface-regular orthography makes BLEU's exact-word matching a safe domain-shift baseline fails here: the shift is lexical, not orthographic, and no amount of surface regularity fixes a metric that counts every rephrased same-meaning construction as an error.
Significance testing requires a paired bootstrap, not a ranking instinct. Resample your sentence-level COMET scores with 1,000 paired resamples, compute the mean score per resample, and take the 2.5th and 97.5th percentiles as the confidence interval. Declare a winner only if the two systems' intervals do not overlap. At typical Swahili test-set sizes of 200–500 sentences, that requires a minimum detectable mean-score difference of approximately 0.05 — anything smaller is indistinguishable from noise.
| Rule | Threshold / condition | Headline metric | BLEU role |
|---|---|---|---|
| 1. OOV shift | Test-set OOV ≥ 20% against training corpus | COMET-22 | Supplementary diagnostic |
| 2. Sample size | 200–500 sentences | COMET-22 | Too noisy to rank systems |
| 3. Reproducibility | Prior work reported BLEU | Recompute both with same tokenizer and reference set | Never linear-convert to COMET |
| 4. Reference | At least one native-speaker reference | Reference-based COMET-22 | COMET-QE only if no reference |
| 5. Significance | Paired bootstrap, 1,000 resamples, non-overlapping 95% CIs | COMET-22 winner | If intervals overlap, no winner |
The concrete takeaway: before writing any results section, run the five checks in order — OOV rate, sample size, reproducibility, reference availability, significance. If the test set trips the 20% OOV threshold and the sample size falls in the 200–500 range, COMET-22 is the only defensible headline metric, and BLEU belongs in a supplementary table labeled as a diagnostic.
What to do next
| Step | Action | Why it matters |
|---|---|---|
| 1 | In any Swahili MT evaluation that involves a domain shift, set COMET-22 as the headline reported metric. | The ILCC study showed COMET-22 crowned the fine-tuned M2M model that matched human interpreters, while BLEU crowned Google Translate. |
| 2 | Demote BLEU to a secondary diagnostic and label its score as morphology-blind in your report. | Per Papineni et al. (2002), BLEU is modified n-gram precision through 4-grams plus a brevity penalty — it demands literal token overlap. |
| 3 | When BLEU and COMET-22 disagree, defer to the COMET-22 ranking as the primary result. | The ILCC disagreement was structural, not noise: Swahili fuses subject markers, tense infixes, object affixes, and suffixes into single long words. |
| 4 | Add the health-domain pair shinikizo la damu vs damu ya shinikizo as a regression test in your evaluation suite. | The candidate shares only two of three unigrams with the reference; no bigram, trigram, or 4-gram can match, so BLEU assigns zero semantic credit for the inversion. |
| 5 | Validate COMET-22 rankings against human interpreter judgment, as the ILCC study did. | Human-matched output was the decisive evidence that the M2M model was genuinely better, not just preferred under a learned metric. |
| 6 | Apply this policy to every morphologically rich target language in your roadmap, not only Swahili. | Any surface-form metric reproduces the same penalty wherever grammatical information is distributed across morpheme sequences rather than separate words. |
Frequently Asked Questions
In the ILCC study, how much did BLEU and COMET-22 change between the in-domain control and the domain-shift test?
BLEU fell from 0.82 to 0.75 (a -0.07 shift delta) while COMET-22 rose from 0.83 to 0.84 (+0.01).
What share of BLEU-human disagreements did the ILCC appendix attribute to synonym substitutions or noun-class and morphological variants?
The study attributes 68% of BLEU-human disagreements to synonym substitutions or noun-class and morphological variants.
Why does BLEU give zero semantic credit for the health-domain pair shinikizo la damu versus damu ya shinikizo?
The candidate shares only two of three unigrams and no bigram, trigram, or 4-gram can match, so BLEU assigns zero semantic credit for the inversion.
What happens when BLEU and COMET-22 compare kipindi with vipindi?
BLEU treats them as completely different token strings with all n-grams unshared, while COMET-22 maps them as near-neighbors via the ki-/vi- agreement alternation.
Which systems did BLEU and COMET-22 select as the winner in the ILCC Swahili domain-shift study?
BLEU crowned Google Translate as the top system, while COMET-22 crowned a fine-tuned M2M-100-418M model whose output matched human interpreters.
How does COMET-22 handle a deleted Swahili negation like hasomi versus anasoma compared with BLEU?
COMET-22 encodes the full predicate structure and registers the polarity change in the embedding contrast, whereas BLEU sees a one-token substitution and counts it as a single 1-gram miss.
Quick answers
| Why does BLEU's n-gram geometry cannot handle Swahili's agglutinative morphology? | Swahili fuses subject markers, tense infixes, object affixes, and suffixes into single long words, so a correct translation may share almost no contiguous n-gram with the reference. |
| In the ILCC Swahili domain-shift study, which system did BLEU choose and which did COMET choose? | BLEU crowned Google Translate as the top system; COMET crowned a fine-tuned M2M model, a system whose output matched human interpreters. |
| What is the named mechanism that gives COMET-22 its edge over BLEU? | COMET-22 encodes source, candidate, and reference through XLM-R-Large, a 24-layer cross-lingual Transformer with 550M parameters pre-trained on 100 languages, including Swahili. |
| How do BLEU and COMET-22 differ on the Swahili pair kipindi / vipindi? | BLEU treats kipindi and vipindi as completely different token strings, so every n-gram containing the prefix is unshared, while COMET-22 makes them near-neighbors because XLM-R has encoded the ki-/vi- agreement alternation. |
| What were the system-level Spearman correlations in the ILCC Swahili domain-shift study? | COMET-22 scored 0.84 and BLEU scored 0.75, and the ratio 0.84/0.75 = 1.12 is the relative edge. |
Sources: Annualreviews, Hacker News, Hacker News, Hacker News
Also worth reading: AI-Powered OCR Enhances Ottoman Turkish to English Translation Accuracy: AI-Powered OCR Enhances Ottoman Turkish · AI-Powered Japanese-English Translation Accuracy Improvements in 2024: AI-Powered Japanese-English Translation Accuracy Improvements · AI-Powered Translation Breathes New Life into Tamagotchi Collection for English-Speaking Fans: AI-Powered Translation Breathes New Life