Factual Consistency Scoring

The foundational step where a generated summary is broken down into its smallest, indivisible factual claims. Each atomic fact is a self-contained statement that can be independently verified against the source text. This granular approach prevents a single hallucination from being masked by an otherwise fluent paragraph. The process often uses few-shot prompted LLMs to extract a list of stand-alone assertions, enabling high-resolution scoring of factual precision.

The core computational engine for scoring. A fine-tuned NLI model classifies the relationship between a source passage (premise) and a generated atomic fact (hypothesis) into three categories:

• Entailment: The source text directly supports the fact.
• Contradiction: The source text directly refutes the fact.
• Neutral: The source text does not provide enough information. This provides a structured, probabilistic basis for calculating consistency metrics rather than relying on simple string matching.

A specialized scoring function that applies a heavy negative weight to detected contradictions. Unlike a simple accuracy score, a consistency metric must penalize false statements more severely than unsupported ones. A hallucination that directly contradicts the source is a critical failure. Scoring models often use a weighted F1 or a custom cost function where contradiction carries a -1 weight, neutral a 0, and entailment a +1, ensuring that a single false claim can zero out a score.

Advanced scoring goes beyond sentence-level NLI to evaluate cross-sentence consistency. This involves verifying temporal consistency—ensuring events are placed in the correct chronological order and that time-bound facts (e.g., 'the current CEO') are valid for the referenced period. It also checks for logical contradictions where two non-adjacent statements in a summary cannot both be true simultaneously, even if each is individually supported by a different part of the source.

A sub-metric that measures not just if a fact is true, but if it is correctly linked to its provenance. Attribution fidelity verifies that a citation points to the exact passage that entails the claim. A high consistency score requires that the evidence for a fact is present in the specified source segment, penalizing 'gratuitous' citations that reference a relevant but non-evidential document. This is critical for RAG systems where citation accuracy is paramount.

The final stage combines atomic scores into a single, interpretable metric. Common formulations include:

• Factual Precision: The ratio of entailed facts to total generated facts.
• Hallucination Rate: The ratio of contradicted facts to total generated facts.
• FActScore: A specific implementation that computes precision per entity. These aggregate scores provide a single, actionable number for evaluating model trustworthiness in production.

The process of breaking down complex generated text into minimal, self-contained, and indivisible assertions. Each atomic fact represents a single verifiable claim, enabling fine-grained evaluation.

• Enables precise contradiction detection at the clause level
• Forms the input unit for Natural Language Inference (NLI) models
• Example: 'The Eiffel Tower, built in 1889, is in Paris' decomposes into two atomic facts

A core NLP task that determines the logical relationship between a premise (source text) and a hypothesis (generated statement). The output is classified as entailment, contradiction, or neutral.

• Serves as the primary computational engine for automated fact-checking
• Modern NLI models fine-tuned on datasets like ANLI and VitaminC provide robust scoring
• Directly powers the calculation of factual consistency rates

A fine-grained evaluation metric that calculates factual precision by decomposing a generated biography into atomic facts and independently verifying each against a trusted knowledge source like Wikipedia.

• Provides a human-correlated, interpretable numerical score
• Penalizes hallucinations while rewarding supported claims
• Foundational metric for benchmarking model factuality in long-form generation

A measure of uncertainty that clusters token-level predictions by their semantic meaning before calculating entropy. This distinguishes between lexical variation (different words, same meaning) and genuine factual indecision.

• High semantic entropy strongly correlates with hallucination risk
• Used as a predictive signal for when to trigger verification loops
• More robust than raw token probability for detecting confabulation

A prompting technique where a language model first drafts a response, then generates a series of independent fact-checking questions to systematically verify and correct its own initial output.

• Implements a self-critique loop without external tools
• Reduces hallucination rates by forcing explicit evidence review
• Effective for improving consistency in zero-shot scenarios

A metric evaluating how accurately a generated statement's citations point to the specific source passages that directly support it. It ensures references are not just topically relevant but precisely evidential.

• Critical for RAG system evaluation beyond simple relevance
• Measures whether citation X genuinely proves claim Y
• Prevents 'citation laundering' where plausible but unsupportive sources are linked