Source Citation Recall

Source Citation Recall measures the proportion of source statements or facts used in a generated answer that are correctly attributed to their originating documents. It is calculated as:

(Number of Correctly Attributed Facts) / (Total Number of Verifiable Facts in the Answer)

• A fact is a discrete, atomic piece of information derived from the source context.
• Attribution is typically a direct citation (e.g., [Doc1]) or an indirect reference that can be unambiguously traced to a specific source passage.
• A score of 1.0 indicates perfect attribution; a score of 0.0 indicates no facts are correctly cited, even if the answer is factually correct.

While Source Citation Recall measures attribution completeness, Source Citation Precision measures attribution accuracy. They are complementary metrics that form a complete picture of citation quality.

• Recall (Completeness): "Of all the facts in the answer, how many were cited?" High recall minimizes unattributed claims.
• Precision (Accuracy): "Of all the citations provided, how many were correct?" High precision minimizes incorrect or hallucinated citations.

An ideal RAG system optimizes for both. A system can have high recall but low precision (cites everything, but many citations are wrong) or high precision but low recall (citations are accurate but many facts are left uncited).

Source Citation Recall is a stricter, more granular subset of the broader Answer Faithfulness metric.

• Answer Faithfulness asks: "Is the entire answer supported by the provided context?" It's a holistic, often binary or graded judgment.
• Source Citation Recall asks: "For each specific factual claim within the answer, can you point to its exact source?" It requires traceability.

A high Source Citation Recall score is strong evidence for high Answer Faithfulness, as it provides an audit trail. However, it is possible for an answer to be faithful (all info is in the context) but have low citation recall if the model fails to explicitly link claims to sources.

Measuring Source Citation Recall requires structured evaluation, often involving:

• Atomic Fact Decomposition: Breaking the generated answer into individual, verifiable propositions (e.g., "The project launched in 2023," "It used a transformer architecture").
• Source Provenance Mapping: For each atomic fact, determining if it exists in the retrieved source documents and identifying the specific passage.
• Automated Methods: Using Named Entity Recognition (NER) and Semantic Similarity (e.g., Sentence-BERT) to align answer claims with source sentences. Frameworks like RAGAS and TruLens implement automated approximations.
• Human-in-the-Loop: For high-stakes applications, human annotators perform the final mapping to establish a ground-truth benchmark for automated scorers.

High Source Citation Recall is a primary engineering defense against model hallucination in RAG systems. It enforces a discipline of provenance.

• Attribution Pressure: Designing the system to require citations for key claims forces the generator to ground its output.
• Debugging Aid: When a hallucination occurs, low citation recall for that fact immediately flags it as unsupported, speeding up root-cause analysis (e.g., was the fact missing from retrieval, or did the generator ignore the source?).
• User Trust: Answers with clear, comprehensive citations allow users to verify information, increasing confidence in the system's outputs, especially in domains like legal, medical, and financial analysis.

Achieving high Source Citation Recall presents specific engineering challenges:

• Fact Fusion: The generator synthesizes information from multiple sources into a single sentence. Should this be one citation or several? Recall scoring must handle multi-source facts.
• Paraphrasing & Summarization: The answer paraphrases a source. Automated metrics must recognize semantic equivalence beyond lexical overlap.
• Implicit Knowledge: The generator adds universally true or common-sense facts not explicitly in the sources (e.g., "Paris is the capital of France"). Evaluators must decide if this requires a citation.
• Noise in Retrieval: If retrieved documents contain conflicting or subtly incorrect data, the generator may correctly cite a source that is itself wrong. This highlights the need for Source Citation Precision alongside recall.

A custom evaluation pipeline can be built using sentence embeddings to compute recall. This method involves breaking the answer and source documents into claims and measuring semantic overlap.

• Steps:
  1. Decompose the gold-standard source documents and the generated answer into atomic factual statements.
  2. Generate embeddings for each statement using a model like Sentence-BERT.
  3. For each source statement, compute if there is a semantically similar (cosine similarity above a threshold) statement in the answer.
• Metric: Source Citation Recall = (Matched Source Statements) / (Total Source Statements).
• Advantage: Provides a deterministic, model-agnostic score less prone to judge LLM variability.

The counterpart to recall, Source Citation Precision measures the accuracy of the citations themselves. It calculates the proportion of citations in a generated answer that correctly and accurately reference the source of the stated information.

• High precision, low recall: The answer cites few sources, but those citations are correct. Information may be missing citations.
• Balanced with recall: A system with high scores in both metrics produces answers where most information is cited (high recall) and those citations are accurate (high precision).

Answer Faithfulness (or Factual Consistency) measures the extent to which a generated answer is factually consistent with and supported by the provided source context. It is a prerequisite for accurate citation.

• Core Question: Does the answer contain any statements that cannot be inferred from the provided sources?
• Relationship to Citation: A faithful answer can be fully supported by citations, making high citation recall possible. An unfaithful answer contains hallucinations, which by definition cannot be cited.

A Grounding Score evaluates the degree to which a model's generated output is substantiated by specific, attributable information from its provided source materials. It is a broader concept that often encompasses both faithfulness and citation quality.

• Implementation: Can be measured by cross-referencing generated claims with source text, often using Natural Language Inference (NLI) models or entailment classifiers.
• Key Difference: While Citation Recall/Precision check for explicit citation markers, grounding scores evaluate the implicit factual support, which is crucial for evaluating answers where citations are not formatted.

Context Relevance assesses the degree to which the text passages retrieved and provided to the LLM are pertinent and useful for answering the specific query. It is an upstream metric that directly impacts citation quality.

• Cause and Effect: If retrieved context is irrelevant (low context relevance), the LLM cannot generate a well-supported answer, leading to low citation recall and potential hallucinations.
• Measurement: Often scored by having an LLM judge, on a scale, whether a retrieved passage contains necessary information to answer the query.

In the retrieval stage, Retrieval Recall measures the proportion of all relevant documents in a corpus that are successfully retrieved for a given query. This is a foundational metric for the entire RAG pipeline.

• Direct Impact: Low retrieval recall means relevant source documents never reach the LLM, placing a hard ceiling on the potential Source Citation Recall. The LLM cannot cite what it never sees.
• Optimization Target: Improving retrieval recall (via better embedding models, hybrid search, or chunking strategies) is often the first step to improving end-to-end citation performance.