Citation Recall

Citation Recall is calculated as the ratio of correctly supported claims to the total number of factual claims in a generated text. A claim is 'supported' if the cited source genuinely contains the information asserted. This metric is the direct counterpart to Citation Precision, which measures the proportion of provided citations that are valid. Together, they form the F1 score for legal grounding.

The metric is formally defined as:

• Citation Recall = (Number of claims with a correct supporting citation) / (Total number of factual claims in the output)

A high recall score indicates the model rarely makes an unsupported assertion. A low score signals frequent hallucination or the model making claims without even attempting to provide authority, a critical failure in legal reasoning.

It is vital to distinguish Citation Recall from Citation Precision:

• Recall punishes the model for making a factual claim with no citation at all, or with a citation that does not support it.
• Precision punishes the model for providing a citation that is irrelevant, fabricated, or does not support the specific claim it's attached to.

A perfect legal AI must maximize both, ensuring every assertion is sourced and every source is correct.

Citation Recall is a primary guardrail in Retrieval-Augmented Generation (RAG) pipelines for law. After a model generates a response grounded on retrieved documents, a Natural Language Inference (NLI) model can automatically verify each claim against its cited source. Claims that fail this entailment check lower the recall score and can be flagged for human review or automatic redaction.

The LegalBench benchmark includes tasks specifically designed to test a model's ability to correctly cite authority. These tasks evaluate whether a model can distinguish between a holding from a real case and a plausible-sounding but fabricated one. Performance on these benchmarks provides a standardized Citation Recall score, allowing for direct comparison between different legal AI systems.

Automated metrics can be noisy. A robust evaluation protocol involves domain-expert human review:

• A legal professional is presented with the generated text and its citations.
• They verify if each cited source exists and genuinely supports the specific proposition.
• This gold-standard judgment is used to calculate the true Citation Recall and to fine-tune the automated verifier models that will eventually replace the human in the loop.

The complementary metric to Citation Recall, measuring the proportion of provided citations that genuinely support the associated claim. While Recall asks 'are all claims supported?', Precision asks 'are all citations valid?'. A system with high Recall but low Precision may cite irrelevant or fabricated sources. Key considerations:

• Detects hallucinated references that look plausible but don't exist
• Penalizes irrelevant citations that point to correct sources but don't support the specific claim
• Together with Recall, forms the F1 score for citation integrity
• Critical for catching the most dangerous failure mode: confident assertions backed by fake authority

A granular metric that quantifies the degree to which a generated statement can be directly linked to a specific segment of a source document. Unlike binary citation checks, Attribution Scoring measures the strength of the evidentiary connection. Implementation approaches:

• Span-level alignment: Maps generated text spans to source text spans
• Entailment probability: Uses NLI models to score how strongly a source entails a claim
• Attention-based tracing: Analyzes model attention weights to track information provenance
• Provides a continuous score rather than a binary supported/unsupported verdict

The automated process of verifying that every factual claim in a generated text is explicitly supported by the provided source document. Serves as a critical guardrail in legal AI pipelines. Operational workflow:

• Decomposes generated text into atomic factual claims
• Retrieves the most relevant source passages for each claim
• Applies Natural Language Inference (NLI) to determine if each claim is entailed, contradicted, or neutral
• Flags unsupported claims for human review or automatic revision
• Often implemented as a verifier model that runs asynchronously before output delivery

The capability of an AI system to pinpoint the exact origin of information in its output, providing a transparent audit trail from conclusion back to raw source text. Goes beyond citation to enable full provenance tracking. Essential features:

• Granular references: Links claims to specific paragraphs, pages, or line numbers
• Multi-document tracing: Tracks information synthesized across multiple sources
• Confidence indicators: Shows the system's certainty about each attribution
• Human-verifiable format: Presents attributions in a way that enables rapid manual validation
• Forms the foundation for contestable AI systems where every assertion can be challenged and traced

A quantitative evaluation framework that measures the factual consistency of a generated summary or answer relative to the source material. Identifies contradictions and unsupported fabrications. Common approaches:

• Entailment-based: Uses NLI to check if the output is logically entailed by the source
• Question-answering based: Generates questions from the output and checks if the source can answer them
• Overlap-based: Measures n-gram and semantic similarity between output and source
• Contradiction detection: Specifically identifies statements that conflict with the source
• Legal applications require zero-tolerance for contradiction due to the adversarial nature of legal proceedings

A faithfulness metric that evaluates whether a model's response is strictly derived from the user-provided context, penalizing the introduction of external knowledge or assumptions not present in the input. Why it matters in legal AI:

• Prevents models from filling gaps with parametric knowledge that may be outdated or incorrect
• Ensures analysis is limited to the specific documents under review
• Critical for discovery and due diligence where scope must be controlled
• Measured by comparing the output against both the provided context and the model's behavior when context is removed
• A model with perfect Context Adherence should produce identical outputs regardless of its training data when given the same context