BLEU Score

BLEU operates by counting the number of n-gram matches between the candidate and reference translations. It computes precision for unigrams (single words), bigrams (two-word sequences), trigrams, and quadgrams. The core insight is that higher-order n-grams capture fluency and grammatical structure, while unigrams assess adequacy of word choice. A clipped count mechanism prevents a candidate from gaming the score by repeating a correctly matched word excessively.

Because precision-based metrics favor overly short translations, BLEU applies a brevity penalty (BP). If the candidate translation is shorter than the reference, the score is multiplied by a factor that decays exponentially. This penalizes candidates that omit content to artificially inflate precision. If the candidate length equals or exceeds the reference, the penalty is 1.0, meaning no penalty is applied. This ensures recall is implicitly considered.

BLEU was originally designed as a corpus-level metric, meaning it calculates a single score over a large test set of thousands of sentences. It performs poorly at the sentence level because a single sentence may have zero n-gram matches with a reference, yielding a score of zero even if the translation is semantically correct. For sentence-level evaluation, neural metrics like COMET are preferred.

BLEU supports comparison against multiple human reference translations to account for the inherent variability in valid translations. A candidate n-gram is considered a match if it appears in any of the references. Using multiple references significantly increases the correlation between BLEU and human judgment, as it captures a wider range of acceptable phrasings and synonyms.

The final BLEU score is the geometric mean of the modified n-gram precisions (from unigram to quadgram), multiplied by the brevity penalty. Using a geometric mean ensures that a candidate must perform well across all n-gram orders to achieve a high score. A catastrophic failure at any single order (e.g., zero bigram matches) drives the entire score toward zero, enforcing holistic translation quality.

BLEU has well-documented weaknesses. It ignores synonymy and paraphrasing, penalizing valid translations that use different wording. It treats all words equally, failing to distinguish between function words (e.g., 'the', 'a') and content words (e.g., 'surgery', 'inflation'). It also cannot assess semantic equivalence or grammatical correctness beyond surface-level n-gram overlap. These limitations motivated the development of neural metrics like BERTScore and COMET.