Citation Precision

Citation Precision is the ratio of correctly supported citations to the total number of citations provided by a model. It answers: "Of all the sources the model cited, how many actually back up the specific claim?"

• Formula: Precision = Correct Citations / Total Citations
• A score of 1.0 means every cited source perfectly supports its associated claim
• A score of 0.5 means half the citations are irrelevant, misleading, or contradictory
• Distinct from Citation Recall, which measures whether claims have citations at all

Citation Precision can be measured at multiple levels of granularity, each revealing different failure modes:

• Passage-Level: Does the cited document section broadly relate to the claim?
• Sentence-Level: Does the specific cited sentence directly support the assertion?
• Span-Level: Does the exact text span pointed to by the citation contain the supporting evidence?
• Claim Decomposition: Breaking a generated statement into atomic claims and verifying each independently against its citation

Span-level evaluation is the most rigorous, catching cases where a model cites a relevant document but points to the wrong paragraph within it.

Modern Citation Precision measurement relies heavily on Natural Language Inference (NLI) to automate verification at scale:

• Each cited passage and its associated claim form an NLI premise-hypothesis pair
• The NLI model classifies the relationship as entailment, contradiction, or neutral
• Only entailment counts as a correct citation for precision calculation
• Contradiction citations are particularly dangerous—they actively undermine trust
• Neutral citations indicate the source is topically related but doesn't substantiate the claim

This approach enables automated, reproducible evaluation without requiring human annotators for every citation pair.

Citation Precision degrades through several predictable failure patterns:

• Source-Content Mismatch: Citing a document that discusses the right topic but doesn't contain the specific fact claimed
• Hallucinated Citations: Generating references to papers, URLs, or legal cases that don't exist—a severe precision failure
• Misattributed Evidence: Correctly citing a real source but attributing a finding to it that actually came from a different source
• Overgeneralization: Citing a narrow study to support a broad, sweeping claim the research doesn't justify
• Temporal Mismatch: Citing outdated sources for claims that require current information

Each failure mode requires different mitigation strategies, from improved retrieval to stricter verification loops.

Several specialized benchmarks assess Citation Precision in long-form generation:

• FActScore: Decomposes biographies into atomic facts and verifies each against Wikipedia, measuring the percentage of supported facts
• RAGTruth: Evaluates hallucination in RAG systems at both passage and word levels across multiple domains
• Attributed QA: Tests whether models can correctly link answers to specific evidence spans in provided documents
• ALCE (Automatic LLMs' Citation Evaluation): Benchmarks citation quality across question types requiring different evidence retrieval strategies

These benchmarks reveal that even state-of-the-art models often achieve Citation Precision below 70-80% on complex tasks.

Improving Citation Precision requires interventions at multiple stages of the generation pipeline:

• Post-hoc Verification: Using a separate NLI model to check each citation-claim pair before presenting output to the user
• Chain-of-Verification (CoVe): Prompting the model to generate verification questions about its own claims and re-check cited sources
• Constrained Decoding: Restricting generation to only produce claims that can be directly mapped to retrieved evidence spans
• Citation-Aware Training: Fine-tuning models on datasets like FaithDial where hallucinated citations have been corrected
• Retrieval Quality Improvement: Ensuring the retriever fetches highly relevant passages, as poor retrieval guarantees poor citation precision

The proportion of generated claims that are supported by a cited source. While Citation Precision measures the relevance of provided citations, Citation Recall measures the model's completeness in providing evidence for all statements it makes.

• Formula: (Number of cited claims) / (Total number of verifiable claims)
• High Recall: The model rarely makes unsupported assertions.
• Trade-off: Often inversely correlated with precision; a model citing everything may include irrelevant sources.

A composite metric evaluating whether a model can correctly link a generated claim to the specific segment of a source document that supports it. It is often measured by the harmonic mean of Citation Recall and Citation Precision.

• Granularity: Requires segment-level or passage-level alignment, not just document-level.
• Use Case: Critical for evaluating Retrieval-Augmented Generation (RAG) systems where provenance is paramount.
• Benchmarking: Datasets like RAGTruth are specifically designed to test this capability.

A metric evaluating whether all factual claims in a generated text are supported by a source document. It measures the alignment between the output and the provided grounding context, distinct from citation precision which focuses on the relevance of the cited evidence itself.

• NLI-Based: Often evaluated using Natural Language Inference models to detect contradictions.
• Scope: Confined to faithfulness to the provided source, not external world knowledge.
• Error Types: Detects intrinsic hallucinations where the output contradicts the input.

A method for assessing factual accuracy by framing the relationship between a source text and a generated hypothesis as a Natural Language Inference task. It classifies the relationship as entailment, contradiction, or neutral.

• Entailment: The source text logically implies the generated claim (supported).
• Contradiction: The source text logically negates the generated claim (hallucination).
• Neutral: The source text does not provide enough information to verify the claim (potential extrinsic hallucination).

A composite metric that calculates the harmonic mean between the precision and recall of factual knowledge units extracted by a model. It balances exactness and completeness in information extraction.

• Factual Precision: Ratio of correct factual statements to total factual statements generated.
• Factual Recall: Ratio of correct factual statements to total factual statements in the ground-truth source.
• Application: Used to evaluate summarization and data-to-text generation systems holistically.

A human-aligned evaluation metric that breaks a long-form generation into atomic facts and verifies each against a trusted knowledge base like Wikipedia. It calculates the percentage of supported facts.

• Decomposition: Splits complex sentences into individual, verifiable claims.
• Knowledge Base: Relies on a static, curated corpus for verification.
• Limitation: Cannot verify claims about events or entities not present in the knowledge base, leading to potential false negatives.