BERTScore

BERTScore leverages contextual embeddings from pre-trained transformer models like BERT, RoBERTa, or DistilBERT. Each token in the candidate and reference text is represented as a vector that captures its meaning within the specific sentence context, not just its static dictionary definition. This allows the metric to recognize that 'automobile' and 'car' are highly similar in a given context.

• Token Matching: Computes cosine similarity between every candidate token and every reference token.
• Greedy Alignment: Each token in the candidate text is matched to the most similar token in the reference text, and vice versa, to compute precision, recall, and F1.

BERTScore decomposes semantic similarity into three standard metrics, providing a nuanced view of generation quality. Recall measures how much of the reference content is captured by the candidate, penalizing omissions. Precision measures how much of the candidate content is relevant to the reference, penalizing hallucinations or extraneous information. F1 is the harmonic mean, providing a single balanced score.

• Recall-Oriented: Critical for summarization tasks where coverage of key information is paramount.
• Precision-Oriented: Critical for tasks where conciseness and factual accuracy are paramount.

To prevent common words from dominating the similarity score, BERTScore can incorporate Inverse Document Frequency (IDF) weighting. Tokens that appear frequently across a corpus (like 'the' or 'is') receive a lower weight, while rare, domain-specific terms receive a higher weight. This ensures the metric focuses on semantically meaningful words.

• Optional Tuning: IDF weighting can be enabled or disabled based on the evaluation domain.
• Corpus-Specific: Weights are computed from a representative background corpus, making the metric adaptable to specialized jargon.

Raw BERTScore values are often compressed into a narrow range (e.g., 0.95–1.0), making it difficult to distinguish between good and excellent systems. The metric applies a baseline rescaling technique that normalizes scores against a lower-bound baseline (e.g., random or constant outputs). This stretches the score distribution, making the final F1 value more interpretable and comparable to human judgment scales.

• Linear Transformation: Maps raw scores to a more intuitive 0–1 range.
• Improved Discriminability: Makes it easier to see meaningful performance differences between models.

The performance of BERTScore is directly tied to the underlying embedding model. Using a larger, more sophisticated model like RoBERTa-large yields higher correlation with human judgments than a smaller model like DistilBERT. The metric allows practitioners to select the model tier that balances computational cost against evaluation accuracy.

• DistilBERT: Fast, low-resource, good for rapid prototyping.
• RoBERTa-large: State-of-the-art correlation, higher computational cost.
• DeBERTa: Advanced model with disentangled attention, often achieving top human correlation scores.

A core advantage over lexical metrics like BLEU or ROUGE is BERTScore's inherent understanding of synonymy and paraphrasing. Because it operates in a continuous semantic space, a candidate that uses different words to express the same meaning as the reference will receive a high score. This makes it ideal for evaluating abstractive summarization and creative text generation where rigid n-gram overlap fails.

• Example: 'The cat sat on the mat' vs. 'A feline rested on the rug' would score poorly on BLEU but highly on BERTScore.
• No Exact Match Required: Evaluates the underlying meaning, not the surface form.