ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

ROUGE measures recall by counting the overlapping sequences of words (n-grams) between a candidate summary and a reference. ROUGE-1 scores unigram overlap (single words), while ROUGE-2 scores bigram overlap (word pairs), capturing basic fluency. ROUGE-L identifies the longest common subsequence, accounting for sentence-level structure without requiring consecutive matches.

Unlike precision-focused metrics, ROUGE prioritizes recall—how much of the reference summary's content the candidate captures. This design choice stems from the observation that human summarizers prioritize content coverage. The formula is:

• ROUGE-N Recall = (Number of overlapping n-grams) / (Total n-grams in reference)
• Precision and F1 variants are also computable for balanced evaluation.

ROUGE is not a single metric but a family:

• ROUGE-1: Lexical coverage, useful for content salience
• ROUGE-2: Captures phrase-level coherence
• ROUGE-L: Longest Common Subsequence, handles non-contiguous matches
• ROUGE-SU4: Skip-bigram with unigrams, allows gaps up to 4 words
• ROUGE-W: Weighted LCS, favors consecutive matches

ROUGE relies on surface-form lexical matching and cannot assess semantic equivalence. In legal summarization, two statements can be factually identical but share zero n-grams (e.g., 'The court found for the plaintiff' vs. 'Judgment was entered in favor of the claimant'). This makes ROUGE a necessary but insufficient metric for legal AI. It must be paired with factual consistency checks using NLI or human evaluation.

ROUGE scores are only as reliable as the human-written reference summaries they compare against. A single reference cannot capture all valid ways to summarize a document. Best practices include:

• Using multiple reference summaries per document to increase score stability
• Reporting 95% confidence intervals via bootstrap resampling
• The official ROUGE-1.5.5.pl Perl script remains the standard implementation for reproducible research

Empirical studies show ROUGE-2 and ROUGE-L correlate reasonably well with human assessments of summary quality for news articles. However, correlation degrades significantly for:

• Abstractive summaries that paraphrase heavily
• Long documents where salient content is dispersed
• Domain-specific texts like legal opinions where terminology precision matters more than lexical overlap In legal AI, ROUGE is best used as a development proxy, not a final quality guarantee.

An automatic evaluation metric that computes the semantic similarity between a candidate summary and a reference by using contextual embeddings from a pre-trained BERT model. Unlike ROUGE's exact n-gram matching, BERTScore leverages cosine similarity between token embeddings to capture paraphrases and synonyms.

• Uses greedy matching to align tokens in the candidate and reference texts
• Provides Precision, Recall, and F1 scores at the token level
• Correlates more strongly with human judgment than n-gram metrics on abstractive summaries
• Particularly effective for legal text where the same concept may be expressed with different terminology

The degree to which a generated summary accurately reflects the stated facts of the source document without contradiction or fabrication. In legal summarization, factual consistency is the paramount quality metric, as a single hallucinated precedent or obligation can have severe consequences.

• Evaluated using Natural Language Inference (NLI) models that classify summary claims as entailed, contradicted, or neutral
• Measured via hallucination rate: the percentage of generated statements not directly supported by the source
• Critical for citation integrity in case law summarization
• Requires source attribution techniques to ground each claim in specific document passages

A method for evaluating summary faithfulness by breaking down a generated text into minimal, self-contained factual claims for individual verification against the source document. Each atomic fact represents a single, indivisible assertion that can be independently validated.

• Decomposes a summary into a list of discrete subject-predicate-object triples
• Each atomic fact is checked against the source using an NLI model
• Provides a fine-grained faithfulness score rather than a holistic judgment
• Enables precise identification of which specific claims in a legal summary are unsupported

The technique of explicitly linking each factual statement in a generated summary back to its precise location in the source document. This transforms a summary from an opaque output into a verifiable, auditable artifact suitable for legal workflows.

• Enables citation verification by mapping claims to specific paragraphs or line numbers
• Supports human-in-the-loop review by allowing rapid validation of AI-generated content
• Often implemented via span highlighting or margin annotations in legal document viewers
• Critical for meeting the high citation integrity standards required in legal practice

An iterative prompting technique for generating increasingly dense and entity-rich summaries without increasing their overall length. Starting from an initial sparse summary, the model repeatedly identifies missing salient entities and fuses them into the existing text.

• Produces summaries with progressively higher information density
• Each iteration adds entities without adding tokens, forcing lexical compression
• Useful for generating headnote-style summaries that pack maximum legal information into minimal space
• Balances the trade-off between coverage and conciseness in multi-document legal synthesis

A task where a model determines if a hypothesis is entailed by, contradicts, or is neutral to a given premise. In summarization evaluation, NLI models serve as automated fact-checkers that verify whether each claim in a summary logically follows from the source text.

• Fine-tuned models like DeBERTa and RoBERTa achieve high accuracy on legal NLI benchmarks
• Used to compute factual consistency scores by treating source text as premise and summary claims as hypotheses
• Detects hallucinations where a summary asserts facts absent from or contradictory to the original document
• Forms the backbone of modern faithfulness evaluation pipelines for legal AI systems