Citation Precision

Citation precision focuses on the accuracy of provided references, not the completeness of citation coverage.

• Precision: Of all citations provided, what fraction are valid and supportive?
• Recall: Of all claims that should be cited, what fraction actually received a citation?

A system with high precision but low recall provides few, highly accurate citations. A system with high recall but low precision floods the text with references, many of which are irrelevant or fabricated. In legal contexts, precision is the non-negotiable floor—a single hallucinated citation destroys credibility faster than a missing one.

Automated citation precision systems replicate the human process of Shepardizing—verifying that a cited authority is still good law and genuinely stands for the proposition asserted.

• The system extracts the pinpoint citation and the legal proposition it allegedly supports
• It queries a ground-truth legal database to retrieve the full text of the cited authority
• A Natural Language Inference (NLI) entailment model determines whether the cited passage actually supports, contradicts, or is neutral to the claim

This process catches not only fabricated citations but also mischaracterized holdings, where a real case is cited but does not actually say what the analysis claims.

Fabricated legal citations exhibit detectable patterns that precision systems are trained to identify.

• Plausible but non-existent volume numbers: A reporter volume that falls within a real range but does not contain that specific page
• Real judge, fictional case name: Combining an actual jurist with a procedurally plausible but non-existent case style
• Mismatched parties and legal principles: A real case cited for a holding completely unrelated to its actual subject matter
• Correct citation format, invalid content: The citation follows Bluebook rules perfectly but points to a void

Detection systems cross-reference court dockets, reporter databases, and legal knowledge graphs to flag these synthetic-but-plausible references before they reach a reader.

The core technical mechanism for citation precision is Natural Language Inference (NLI) entailment classification.

• Premise: The full text of the cited legal authority
• Hypothesis: The specific proposition the generated text claims the authority supports
• Classification: Entailment, Contradiction, or Neutral

A citation is precise only when the NLI model returns Entailment with high confidence. Contradiction indicates the authority actually says the opposite. Neutral means the citation is irrelevant padding. Legal AI systems deploy fine-tuned legal NLI models trained on adversarial examples of mischaracterized holdings to maximize precision.

Beyond validating that a case exists, precision systems must verify pinpoint citations—references to specific pages or paragraphs within an opinion.

• A system may correctly cite Marbury v. Madison, 5 U.S. 137 (1803), but fabricate a pinpoint to page 180 for a proposition about judicial review
• Precision validation extracts the exact text at the pinpoint location and runs entailment against the associated claim
• This catches page-shifting hallucinations where the general case is real but the specific supporting text is invented

Pinpoint validation requires access to full-text judicial opinions with paragraph-level granularity, not just case metadata.

Citation precision is operationalized through rigorous scoring frameworks that produce auditable metrics.

• Citation Precision Score: Number of valid, supportive citations divided by total citations provided
• Strict Precision: Only counts citations where the pinpoint text directly entails the claim
• Lenient Precision: Counts citations where the authority generally supports the proposition, even if the pinpoint is imprecise
• Fabrication Rate: The percentage of citations that reference non-existent authorities

Enterprise legal AI deployments typically require strict precision above 95% and a fabrication rate below 1% before production release. These metrics are continuously monitored in production to detect model drift.

A granular metric that quantifies the degree to which a generated statement can be directly linked to a specific span of text in a source document. Unlike binary verification, attribution scoring provides a confidence spectrum:

• High attribution: The claim is a near-verbatim extraction.
• Low attribution: The claim is a loose paraphrase or synthesis.
• Zero attribution: The claim has no identifiable provenance, flagging a likely hallucination.

The automated process of verifying that every factual claim in a generated text is explicitly supported by the provided source document. This is often implemented using a Natural Language Inference (NLI) model that classifies each claim as entailed, contradicted, or neutral relative to the source. Serves as a real-time guardrail, blocking unsupported assertions before they corrupt a legal memo.

A key performance indicator measuring the proportion of factual claims in a generated legal text that are correctly supported by a citation. Calculated as:

• True Positives: Claim is made and a valid citation is provided.
• False Negatives: Claim is made but no citation is provided, or the citation is irrelevant.

High citation recall indicates a model that consistently provides authority for its assertions, not just plausible-sounding prose.

The end-to-end capability of an AI system to not only generate an answer but also pinpoint the exact origin of every piece of information. This creates a transparent audit trail from a legal conclusion back to the raw source text, paragraph, and line. For a managing partner, source attribution transforms a black-box AI output into a verifiable work product that can be confidently filed with a court.