Citation Recall

Citation Recall is the ratio of cited claims to total verifiable claims in a generated text. It answers: 'Of all the factual statements made, how many did the model attempt to support with a reference?'

• Formula: Number of Claims with a Citation / Total Number of Claims
• A score of 1.0 means every factual assertion has a corresponding citation.
• A low score indicates the model is making unsupported statements, a primary vector for hallucination.

Citation Recall is often paired with Citation Precision, but they measure different things:

• Citation Recall: Are all claims cited? (Completeness of evidence)
• Citation Precision: Are the provided citations actually relevant and supportive? (Correctness of evidence)

A model can have high recall (citing everything) but low precision (citing irrelevant sources). The Knowledge F1 score is the harmonic mean of these two metrics, providing a balanced view of citation quality.

In Retrieval-Augmented Generation (RAG) pipelines, Citation Recall is a critical diagnostic metric. It directly measures the system's ability to ground its output in the retrieved context.

• A drop in recall often signals a failure in the retrieval step or a model ignoring the provided documents.
• It is a key component of benchmarks like RAGTruth, which evaluates hallucination at the passage and word level.
• Engineers use recall to tune retrieval hyperparameters and prompt instructions to force evidence-backed generation.

While related, Citation Recall and Factual Recall are distinct metrics:

• Citation Recall: Measures the presence of a reference link, regardless of whether the claim itself is true.
• Factual Recall: Measures the proportion of true facts from a source document that the model successfully included in its output.

A model can achieve perfect Citation Recall by citing a source for every sentence, yet have poor Factual Recall if it misrepresents what the source actually says. Both are needed for a complete faithfulness assessment.

Modern evaluation frameworks automate Citation Recall measurement using Natural Language Inference (NLI) models:

• The generated text is decomposed into atomic claims.
• Each claim is paired with its cited source segment.
• An NLI model classifies the relationship as entailment, contradiction, or neutral.
• Only entailed claims count toward both recall and precision.

Tools like FActScore and SelfCheckGPT operationalize this process for large-scale, human-aligned evaluation without manual review.

Citation Recall is inversely correlated with hallucination rate. A low recall score is a leading indicator of systemic factual errors:

• Entity-Level Hallucinations: Invented names or numbers often appear in uncited passages.
• Relation-Level Errors: False connections between concepts proliferate when the model is not forced to anchor claims.
• Critical Error Rate: In high-stakes domains like medicine or finance, a recall below 1.0 is unacceptable, as any unsupported claim could cause direct harm.

Implementing Guardrails like NeMo can enforce a minimum recall threshold in production systems.

The complementary metric to Citation Recall, measuring the relevance and correctness of provided citations. While recall tracks whether claims have any cited source, precision evaluates whether the cited source actually supports the specific claim.

• High precision, low recall: Model cites sparingly but accurately
• Low precision, high recall: Model cites aggressively but often irrelevantly
• Both metrics together form the Attribution F1 score

A composite evaluation metric that determines if a model can correctly link a generated claim to the specific segment of a source document that supports it. Combines Citation Recall and Citation Precision into a unified measure of evidentiary integrity.

• Measures end-to-end attribution quality
• Critical for Retrieval-Augmented Generation (RAG) systems
• Used in benchmarks like RAGTruth for word-level and passage-level evaluation

A human-aligned evaluation metric that breaks long-form generations into atomic facts and verifies each against a trusted knowledge base like Wikipedia. Calculates the percentage of supported facts in a generation.

• Developed by researchers at the University of Washington and Meta AI
• Addresses the limitation of single-sentence evaluation for long-form text
• Provides granular, per-fact verification rather than holistic scoring

A method for assessing factual accuracy by framing the relationship between a source text and a generated hypothesis as a Natural Language Inference task. Each claim is classified as:

• Entailment: The source logically supports the claim
• Contradiction: The source refutes the claim
• Neutral: The source provides no information about the claim

Forms the backbone of automated faithfulness metrics like ANLI and SummaC.

An automated evaluation score that determines if a generated summary or response can be logically deduced from the input source without introducing extraneous information. Often implemented using NLI models.

• Distinct from factual accuracy: a statement can be factually true but unfaithful if not grounded in the provided source
• Critical for abstractive summarization and RAG quality assurance
• Measured by tools like SummaC and QuestEval

A composite metric calculating the harmonic mean between the precision and recall of factual knowledge units extracted by a model. Balances exactness and completeness of information retrieval.

• Factual Precision: Ratio of correct generated facts to total generated facts
• Factual Recall: Ratio of correct generated facts to total facts in the ground-truth source
• Provides a single, balanced score for knowledge extraction quality