BLEU (Bilingual Evaluation Understudy)

The core of BLEU is calculating modified n-gram precision. It counts how many n-grams (contiguous sequences of n words) from the candidate translation appear in the reference translations. A crucial modification prevents gaming the score: each n-gram in the candidate is clipped at the maximum number of times it appears in any single reference. This penalizes overly repetitive outputs.

• Unigrams (1-gram): Measure adequacy (are the correct words present?).
• Bigrams (2-gram) and higher: Measure fluency (is the word order natural?).

BLEU includes a Brevity Penalty (BP) to counteract the advantage a very short candidate translation would have in n-gram precision. A short output can achieve high precision by matching a few key n-grams but fail to convey the full meaning.

The penalty is calculated as:

• BP = 1 if the candidate length (c) exceeds the reference length (r).
• BP = exp(1 - r / c) if c <= r.

This exponentially decays the score for candidates shorter than the effective reference length, ensuring outputs must be sufficiently comprehensive.

BLEU does not average the precision scores for different n-gram orders (e.g., 1-gram, 2-gram, 3-gram, 4-gram). Instead, it computes the geometric mean. This is more sensitive to low scores across any n-gram order than an arithmetic mean.

The final BLEU score formula is: BLEU = BP * exp( Σ (w_n * log p_n) ) Where p_n is the modified precision for n-gram of order n, and w_n is a weight (typically 1/4 for n=1 to 4, giving equal weight to each order). Using logarithms transforms the product of precisions into a sum.

BLEU is designed to work with multiple reference translations for a single source sentence. This accounts for the fact that there are many valid ways to translate the same idea. The algorithm compares the candidate against the union of all references.

For example, if references are "the cat sat" and "a cat was sitting," a candidate "the cat was sitting" can match unigrams from both. Using multiple references increases the robustness of the evaluation by covering more lexical and syntactic variations, better approximating human judgment.

BLEU is a corpus-level metric, not a sentence-level metric. The n-gram counts and the brevity penalty are computed over an entire evaluation corpus (typically hundreds or thousands of sentences). While a sentence-level BLEU can be calculated, it is often unreliable and not the intended use.

This design provides a stable, aggregate measure of translation system quality. The score is most meaningful when comparing different systems or versions of a system on the same standardized test corpus (e.g., WMT shared task datasets).

Despite its dominance, BLEU has well-known limitations:

• No semantic understanding: It only measures surface-form overlap. A paraphrase with different words but identical meaning scores zero.
• Poor correlation at sentence level: It was designed for corpus-level evaluation.
• Insensitive to word order for longer n-grams: While bigrams capture some order, significant reordering with the same words can still yield a deceptively high score.
• Lack of recall component: It is precision-oriented. A translation that includes all necessary information but adds extra, correct details may be penalized by the brevity penalty or lower precision.

These limitations have spurred the development of metrics like METEOR, which includes synonym matching and recall, and learned metrics like BERTScore.

Perplexity is an intrinsic evaluation metric that measures how well a probability model (like a language model) predicts a sample. A lower perplexity indicates the model is more confident and accurate in its predictions.

• Calculation: The exponential of the average negative log-likelihood per word.
• Intrinsic vs. Extrinsic: Unlike BLEU (an extrinsic task-based metric), perplexity evaluates the model's fundamental language understanding without a specific task.
• Application: Used to evaluate and compare language models during pre-training and fine-tuning.

Human Evaluation is the process of using human judges to assess the quality of AI-generated text, serving as the ultimate gold standard against which automated metrics like BLEU are validated.

• Common Dimensions: Fluency, adequacy, coherence, and factual consistency.
• Limitations: Expensive, time-consuming, and can suffer from low inter-annotator agreement.
• Role in Benchmarking: Automated metrics like BLEU are optimized to correlate with aggregate human judgments, but they remain proxies with inherent limitations.

Hallucination Rate is a metric quantifying the frequency with which a generative AI model produces confident but factually incorrect or nonsensical output not grounded in its source data or the real world.

• Critical Difference from BLEU: BLEU measures surface-form overlap, not factual correctness. A translation can have a high BLEU score while containing critical factual hallucinations.
• Evaluation Challenge: Requires fact-checking against a knowledge source or reference, often involving natural language inference (NLI) models or human review.
• Key for Agents: A low hallucination rate is a critical SLO for agents performing information retrieval and synthesis.