Factual Consistency Check

Natural Language Inference (NLI) is a core discriminative method that uses a pre-trained model to classify the logical relationship between a generated claim (hypothesis) and a source text (premise). The model outputs one of three labels:

• Entailment: The source text supports the claim.
• Contradiction: The source text explicitly contradicts the claim.
• Neutral: The relationship cannot be determined.

This provides a direct, probability-scored assessment of factual alignment. Models like DeBERTa and RoBERTa, fine-tuned on datasets like MNLI or SNLI, are commonly used for this zero-shot or fine-tuned classification task.

This generative method decomposes a long-form generated text into individual atomic claims. For each claim, a Question Answering (QA) model is used to query the source document, generating an answer from the ground truth. The original claim is then compared to the QA model's answer.

Key steps:

1. Claim Extraction: Isolate factual statements from the summary or answer.
2. Question Formulation: Convert claims into questions (e.g., "The report stated revenue was $5M" becomes "What was the revenue?").
3. Answer Span Retrieval: Use a QA model (e.g., based on BERT) to extract the answer from the source.
4. Similarity Scoring: Compare the original claim and the retrieved answer using lexical (ROUGE) or semantic (BERTScore) metrics.

A low similarity score indicates a potential hallucination.

This approach uses reference-based evaluation metrics to measure the overlap between a generated text and its source, but with a focus on factual rather than lexical similarity.

• BERTScore: Calculates the cosine similarity between the contextual embeddings of tokens in the generated text and the source document. It captures semantic equivalence better than n-gram methods.
• BLEURT: A learned evaluation metric based on BERT that is fine-tuned on human judgments of quality, making it more correlated with factual consistency ratings.
• FactCC: A transformer-based model specifically trained to detect factual inconsistencies in summarization, classifying sentences as consistent or inconsistent with the source.

These methods provide a continuous score, allowing for ranking and threshold-based detection of inconsistencies.

This method validates generated content against a structured knowledge base or enterprise knowledge graph, checking for semantic and relational accuracy.

Process:

1. Entity Linking: Identify named entities (people, places, organizations) in the generated text and link them to canonical nodes in the knowledge graph.
2. Relation Extraction: Identify the stated relationships between entities.
3. Graph Query: Verify if the extracted (subject, predicate, object) triple exists in the knowledge graph.

For example, the claim "Elon Musk founded SpaceX in 2001" would be checked against the graph for the founded relationship between the entities Elon Musk and SpaceX with a foundingDate property. A missing or conflicting relation indicates a factual error. This method is crucial for ensuring correctness in domains with well-defined ontologies.

These reference-free methods use the generative model's own capabilities to check its work, without an external classifier.

• Self-Consistency Sampling: The model generates multiple responses (e.g., 5-10) to the same prompt. The factual consensus across samples is measured. Claims that vary significantly are flagged as less reliable. This leverages the idea that a model is more likely to be consistent when it is factually certain.
• Chain-of-Verification (CoVe): A prompting technique where the model:
  1. Generates an initial response.
  2. Plans a set of verification questions to fact-check its own response.
  3. Answers those questions independently (avoiding bias from the initial answer).
  4. Revises the original answer based on the verification results. This creates an explicit internal audit trail, forcing the model to confront its own potential errors.

A verifier model is a separate, often smaller classifier that is specifically trained to discriminate between factually consistent and inconsistent model outputs. It represents a dedicated discriminative verification pipeline.

Training Process:

1. Dataset Creation: A dataset is built from (source_document, generated_text, label) triples, where the label indicates factual consistency. Data can come from human annotations or synthetic corruption of good summaries.
2. Model Training: A model like a cross-encoder (which jointly processes the source and claim) is trained on this dataset to output a probability score for consistency.
3. Inference: The trained verifier scores new (source, generation) pairs in production.

This method is highly accurate for specific domains but requires significant labeled data. It is often used as a final quality gate before deploying a generative system's outputs.

Quantifying factual consistency requires specialized metrics beyond standard text similarity scores like BLEU or ROUGE.

• Factual Error Rate (FER): The proportion of atomic claims in a generated text that are incorrect or unsupported by the source.
• Precision/Recall for Claims: Treating each verifiable claim as an item for retrieval from the source.
• Benchmark Datasets: Systems are evaluated on curated datasets like FEVER, TRUE, or SummEval, which contain source-claim pairs annotated for entailment or contradiction.
• Natural Language Inference (NLI) Score: Using a model like DeBERTa fine-tuned on MNLI to classify claim-source pairs as entailment, neutral, or contradiction. The entailment rate serves as a primary consistency score.

Different neural architectures are deployed to perform the consistency verification task, each with trade-offs in accuracy and computational cost.

• Cross-Encoders: A discriminative model (e.g., a transformer) that takes the claim and source text concatenated as input and outputs a direct consistency score. High accuracy but computationally expensive for many claims.
• Bi-Encoders: Encode the claim and source separately into dense vector embeddings (e.g., using Sentence-BERT). Consistency is measured by the cosine similarity between embeddings. Faster for retrieval but generally less precise than cross-encoders.
• Generative Verifiers: A separate LLM is prompted to act as a verifier, generating a judgment (e.g., "Supported" or "Not Supported") and often a justification. Flexible but can be brittle and expensive.
• Question Answering (QA) Models: Decompose a claim into questions, use a QA model to extract answers from the source, and compare answers to the original claim text.

A production-grade factual consistency check typically follows a sequential pipeline:

1. Claim Extraction: Parse the generated text into discrete, atomic factual claims using rule-based segmentation or a dedicated claim-splitting model.
2. Source Retrieval (if not provided): For open-domain checks, use a retriever (e.g., dense passage retrieval) to fetch relevant evidence from a knowledge base or corpus.
3. Claim-Source Alignment: For each claim, identify the most relevant sentences or passages within the source document using semantic similarity.
4. Verification Inference: Pass each (claim, aligned source) pair through the chosen verification model (e.g., NLI model, cross-encoder) to obtain a consistency label and score.
5. Aggregation: Roll up per-claim scores into an overall document-level consistency score (e.g., minimum, average, or proportion of supported claims).

Several technical challenges complicate reliable factual consistency checking.

• Linguistic Variation: The model must recognize paraphrases, synonyms, and inferential reasoning; the claim "The capital is Paris" must be matched to a source saying "France's seat of government is in Paris."
• Implicit Claims: Detecting claims not explicitly stated but strongly implied by the generated text.
• Confidence Calibration: Verification models often produce poorly calibrated confidence scores, making it difficult to set a reliable threshold for "supported."
• Commonsense vs. Factual: Distinguishing between a factual error and a plausible commonsense inference not present in the source.
• Scalability: Applying dense verification models to long documents with hundreds of claims is computationally prohibitive, necessitating efficient filtering stages.

Factual consistency checks are a critical feedback mechanism within Retrieval-Augmented Generation (RAG) architectures.

• Post-Generation Validation: The check runs on the final RAG output, flagging responses for human review or triggering automatic regeneration if consistency is low.
• Iterative Retrieval & Generation: The consistency score can guide a retrieve-then-read-then-verify loop, where low-scoring claims trigger a new, broader retrieval to find supporting evidence.
• Training Data Creation: Consistency labels on model outputs can be used to create datasets for fine-tuning the primary generator to be more faithful.
• Source Attribution: A high-quality check should identify which source passage supports each claim, enabling precise citation and auditability.

For high-stakes applications, automated checks are combined with human oversight.

• Triage and Prioritization: The consistency score ranks outputs, allowing human reviewers to focus on the most likely problematic generations.
• Gold-Standard Annotation: Human annotators create labeled datasets for training and calibrating automated verifiers, following guidelines for claim boundaries and entailment judgments.
• Adversarial Example Creation: Experts deliberately craft inputs that cause the generator to hallucinate, testing the limits of the automated checker.
• Failure Analysis: Regular review of cases where the checker failed (false positives/negatives) to iteratively improve the verification model and pipeline.

The overarching process of identifying when a generative model produces factually incorrect, nonsensical, or unsupported content not grounded in its source data or general knowledge. It encompasses a suite of techniques, including factual consistency checks, contradiction detection, and confidence monitoring.

• Goal: Flag outputs that are plausible-sounding but false.
• Scope: Broader than fact-checking; includes detecting 'confabulations' where the model invents details.

A method that uses pre-trained NLI models (e.g., trained on MNLI) to classify the relationship between a generated claim and a source text as entailment, contradiction, or neutral. A 'contradiction' label directly signals a factual inconsistency.

• Mechanism: Treats the source as a premise and the generated claim as a hypothesis.
• Advantage: Leverages robust, general-purpose models without task-specific fine-tuning.

The process of systematically checking the truthfulness of individual statements (claims) against authoritative external sources like knowledge bases (e.g., Wikidata) or verified databases. It's often automated using information retrieval and textual entailment.

• Granularity: Operates at the atomic claim level (e.g., 'Paris is the capital of France').
• Use Case: Critical for high-stakes domains like healthcare and finance where single false claims carry significant risk.

The process of adjusting a model's predicted probability scores (e.g., token logits) so they accurately reflect the true likelihood of a generated statement being correct. A well-calibrated model's confidence is a reliable signal for hallucination detection.

• Problem: Modern LLMs are often overconfident, even when hallucinating.
• Solution: Techniques like temperature scaling or Platt scaling align confidence with accuracy.

Identifies logical inconsistencies either within a single model output (self-contradiction) or between the output and a known source of truth. It is a core sub-task of factual consistency checking.

• Internal: Detects if an answer states 'The event was in 1995' and later says 'It happened last century'.
• External: Detects if a summary says 'The company profited' while the source states 'The company reported a loss'.

A separate, often smaller discriminative model (e.g., a classifier) trained to evaluate the factuality, correctness, or safety of outputs from a primary generative model. It acts as a dedicated quality gate.

• Training: Trained on datasets of (output, source, label) tuples.
• Deployment: Runs inference on the primary model's outputs to score or flag potential hallucinations before delivery to the user.