BLEU

The core of BLEU is the calculation of modified n-gram precision. For n-grams of size 1 to 4 (unigrams, bigrams, trigrams, 4-grams), it counts how many n-grams from the candidate translation appear in any of the reference translations. Crucially, it uses a clipping mechanism: each n-gram in the candidate is counted only as many times as it appears in the single reference translation where it occurs most. This prevents gaming the metric by repeating common words.

• Example: Candidate: 'the the the the the'.
• Without clipping, unigram precision = 5/5 = 1.0 (flawed).
• With clipping (if reference max count for 'the' is 2), clipped count = 2, precision = 2/5 = 0.4.

N-gram precision alone favors short, overly concise translations that match small snippets. The Brevity Penalty (BP) penalizes candidates shorter than the reference. It is calculated as:

BP = 1 if candidate length > reference length BP = exp(1 - (reference_length / candidate_length)) if candidate length ≤ reference length

• If the candidate is shorter, the penalty exponentially decays the score.
• The 'reference length' is typically the length of the closest reference translation for the calculation, not an average, making it sensitive to under-generation.

BLEU combines the modified precisions for n=1,2,3,4 by taking their geometric mean. Using a geometric mean, rather than an arithmetic mean, ensures that a low score in any one n-gram order (e.g., poor fluency in 4-grams) strongly drags down the total. The formula is:

GM = exp( Σ (w_n * log(p_n)) ) where p_n is the precision for n-gram n.

The standard BLEU uses uniform weights: w_1 = w_2 = w_3 = w_4 = 0.25. This mean is then multiplied by the Brevity Penalty to produce the final BLEU score, which ranges from 0 to 1 (often expressed as a percentage).

A key design feature of BLEU is its ability to use multiple human reference translations for a single source sentence. This accounts for the fact that there are many valid ways to translate the same idea. The n-gram matching and clipping are performed against the union of all references.

• This increases the robustness of the score, as a candidate can match phrasing from any reference.
• In practice, using 4 reference translations is common in benchmarks like WMT. The more references used, the more reliable and less variable the BLEU score tends to be.

BLEU is fundamentally a corpus-level metric, not a sentence-level one. All calculations (n-gram counts, clipping, length ratios) are aggregated over the entire test corpus before the final score is computed.

• This provides a stable, aggregate measure of system performance.
• A sentence-level BLEU score is not defined in the original formulation, though heuristics exist. The corpus-level approach avoids giving undue weight to very short or very long sentences.

While foundational, BLEU has well-documented limitations:

• Lack of Semantic Understanding: It measures surface-form overlap, not meaning. A paraphrase with different words scores zero.
• Poor Correlation with Human Judgment at Sentence Level: Its corpus-level design means it correlates poorly with human ratings for individual sentences.
• Insensitivity to Word Order: Bigram and higher-order n-grams capture some local order, but global discourse and syntax errors may be missed.
• Domain and Language Bias: It works best for languages with similar structure to English and in domains with limited lexical variation. It is less reliable for highly creative or technical text.

These limitations spurred the development of metrics like METEOR, BERTScore, and COMET.

Human Evaluation is the process of using human judges to assess the quality of machine-generated text, serving as the ultimate gold standard against which automated metrics like BLEU are validated. Common evaluation dimensions include:

• Adequacy: The extent to which the generated text conveys the same meaning as the reference (semantic fidelity).
• Fluency: The grammatical correctness and naturalness of the text in the target language.
• Comprehensibility: How easily a human can understand the generated output.

While costly and time-consuming, it is essential for final model validation, as automated metrics can be gamed and may not fully capture nuanced aspects of quality.

Perplexity is an intrinsic evaluation metric that measures how well a probability model (like a language model) predicts a sample of text. It is defined as the exponential of the average negative log-likelihood per word.

• Lower is Better: A lower perplexity indicates the model is less "surprised" by the test data and has a better grasp of the language distribution.
• Intrinsic vs. Extrinsic: Unlike BLEU (an extrinsic task-based metric), perplexity evaluates the model's fundamental language modeling capability without a downstream task.
• Common Use Case: It is widely used during language model pre-training and fine-tuning to monitor convergence and select checkpoints, before final evaluation with metrics like BLEU on translation or summarization tasks.