BERTScore

Unlike ROUGE or BLEU, which rely on exact token matches, BERTScore computes similarity using contextualized embeddings from pre-trained transformer models like BERT. Each token in the candidate summary is matched to the most similar token in the reference based on cosine similarity in embedding space. This allows the metric to recognize that 'attorney' and 'lawyer' are semantically equivalent, a critical capability for evaluating abstractive summarization where phrasing may differ entirely from the source.

BERTScore decomposes evaluation into three standard metrics:

• Precision: Measures how many tokens in the candidate summary align with tokens in the reference, penalizing extraneous or irrelevant content.
• Recall: Measures how many tokens in the reference are captured by the candidate, penalizing omissions.
• F1 Score: The harmonic mean of precision and recall, providing a balanced overall quality measure. This decomposition is essential for legal text summarization, where both factual completeness (recall) and conciseness (precision) are paramount.

BERTScore optionally incorporates Inverse Document Frequency (IDF) weighting to emphasize rare, domain-specific tokens over common stop words. In legal texts, terms like 'estoppel', 'tortfeasor', or 'indemnification' carry significant semantic weight. IDF weighting ensures that correct usage of these specialized terms has a proportionally larger impact on the final score, aligning the metric more closely with human judgments of summary quality in high-stakes legal contexts.

Raw BERTScore values are difficult to interpret because cosine similarity between random embeddings is not zero. The metric applies baseline rescaling by computing the average cosine similarity of random token pairs and using it as a floor. The formula (score - baseline) / (1 - baseline) rescales values so that a score of 0 corresponds to random performance and 1 corresponds to perfect alignment. This makes scores more intuitive and comparable across different model architectures and datasets.

BERTScore is not tied to a single model. Evaluators can select from a hierarchy of pre-trained transformer models depending on the required precision and computational budget:

• BERT Base: Standard 110M parameter model, suitable for general evaluation.
• RoBERTa Large: 355M parameter model with improved training dynamics, offering higher correlation with human judgment.
• DistilBERT: A lighter 66M parameter model for faster evaluation with minimal accuracy loss. This flexibility allows legal AI teams to balance evaluation fidelity against infrastructure cost.

Empirical studies demonstrate that BERTScore achieves significantly higher Pearson correlation with human evaluators than n-gram metrics like ROUGE-L or BLEU, particularly for abstractive summaries. In the legal domain, where paraphrasing and semantic preservation are critical, BERTScore's ability to recognize that 'the court held the defendant liable' and 'the defendant was found responsible by the tribunal' convey the same meaning makes it a more reliable proxy for human assessment of summary quality.

The traditional benchmark for summarization evaluation. ROUGE measures lexical overlap by counting matching n-grams, word sequences, and word pairs between a candidate and reference summary. Unlike BERTScore, it operates on exact token matching and cannot recognize synonyms or paraphrases. In legal contexts, this limitation is critical: ROUGE penalizes a summary that correctly replaces 'the plaintiff was awarded damages' with 'the court granted relief to the claimant,' while BERTScore recognizes their semantic equivalence.

The degree to which a generated summary accurately reflects the stated facts of the source document without contradiction or fabrication. In legal AI, this is the non-negotiable requirement. A summary can achieve a high BERTScore by being semantically similar to a reference while still hallucinating a critical date or party name. Factual consistency is often verified using Natural Language Inference (NLI) models that check if each atomic claim in the summary is entailed by the source text.

A granular evaluation method that breaks a generated summary into minimal, self-contained factual claims before verification. For example, 'The court dismissed the appeal on March 3, 2024, citing lack of standing' decomposes into:

• Claim 1: The court dismissed the appeal.
• Claim 2: The dismissal occurred on March 3, 2024.
• Claim 3: The reason was lack of standing. Each claim is individually checked against the source, providing a precise faithfulness score that BERTScore alone cannot guarantee.

The technique of explicitly linking each factual statement in a generated summary back to its precise location in the source document. This transforms a summary from a black-box output into a verifiable artifact. In legal workflows, source attribution enables attorneys to rapidly validate AI-generated summaries by tracing claims to specific paragraphs, page numbers, or line references. Combined with BERTScore's semantic similarity scoring, it creates a complete evaluation pipeline: BERTScore measures quality, attribution ensures trust.

A foundational NLP task where a model determines the logical relationship between a premise (source text) and a hypothesis (summary claim). The three possible labels are:

• Entailment: The hypothesis is true given the premise.
• Contradiction: The hypothesis is false given the premise.
• Neutral: The premise does not confirm or deny the hypothesis. NLI models fine-tuned on legal corpora serve as automated fact-checkers, catching hallucinations that semantic similarity metrics like BERTScore may miss.

An iterative prompting technique that generates increasingly dense and entity-rich summaries without increasing overall length. Starting from an initial sparse summary, the model repeatedly identifies missing salient entities and fuses them into the text. This produces summaries optimized for information density—a key requirement in legal contexts where every word carries weight. BERTScore is particularly well-suited to evaluate these dense summaries because its contextual embeddings capture nuanced entity relationships that n-gram metrics miss.