Factual Consistency

The process of breaking down a generated text into discrete, verifiable factual claims before evaluation. Each atomic fact represents a single piece of information that can be independently verified against the source.

• Granularity: Operates at the clause or proposition level rather than sentence level
• Independence: Each atom must be verifiable in isolation without relying on other generated claims
• Coverage: Ensures no factual claim escapes evaluation by being embedded in complex syntax

Metrics like FActScore use this approach to calculate the percentage of supported atoms, providing a more precise alternative to holistic human judgment.

A method that frames factual consistency as a Natural Language Inference (NLI) task, classifying the relationship between the source text and the generated hypothesis as entailment, contradiction, or neutral.

• Directional: The source must logically entail the generated claim, not merely be consistent with it
• Strictness: Contradiction indicates hallucination; neutral indicates unsupported extrapolation
• Automation: Enables large-scale evaluation using fine-tuned NLI models like ANLI or DeBERTa

This approach underpins metrics such as Faithfulness and Attribution Score, providing a rigorous logical framework for automated fact-checking.

Factual consistency evaluation balances two competing dimensions: Factual Precision (are generated facts correct?) and Factual Recall (are all source facts captured?).

• Factual Precision: Ratio of correct generated facts to total generated facts — penalizes hallucination
• Factual Recall: Ratio of captured source facts to total source facts — penalizes omission
• Knowledge F1: The harmonic mean of precision and recall, providing a single balanced metric

A system with perfect precision but low recall is safe but incomplete; perfect recall with low precision is comprehensive but unreliable. Production systems must optimize for both.

The requirement that every generated factual claim must be explicitly linked to a specific, identifiable span in the source document. This enables citation-level verification.

• Citation Recall: Proportion of claims that have a supporting citation
• Citation Precision: Proportion of citations that actually support their associated claim
• Granularity: Attribution can operate at passage, sentence, or token level

This characteristic is critical for Retrieval-Augmented Generation (RAG) systems, where the source document is the grounding context. Metrics like Attribution Score and benchmarks like RAGTruth specifically evaluate this capability.

A specialized dimension that verifies whether named entities — people, locations, organizations, dates, and numerical values — in the generated text match the source exactly.

• Entity Hallucination: The invention or substitution of entities not present in the source
• Numerical Fidelity: Verification that quantities, percentages, and dates are transcribed accurately
• Coreference Resolution: Ensuring pronouns and references resolve to the correct source entities

Entity-level errors are among the most dangerous hallucinations because they can transform a nearly correct statement into a dangerously false one, particularly in domains like medicine or finance.

A zero-resource verification technique that exploits the probabilistic nature of language models by sampling multiple outputs and measuring their factual agreement. Hallucinated facts tend to be stochastically unstable.

• SelfCheckGPT: Samples multiple responses and checks for inconsistency across generations
• Semantic Entropy: Clusters semantically equivalent outputs before measuring variance
• Assumption: Grounded facts appear consistently across samples; hallucinations vary randomly

This approach requires no external knowledge base or source document, making it applicable to black-box models where internal probabilities are inaccessible.

An automated evaluation score, often using Natural Language Inference (NLI), that determines if a generated summary can be logically deduced from the input source. A faithful output contains no hallucinated content or extraneous information not present in the grounding document.

• Uses entailment classification: source text as premise, generation as hypothesis
• A score of 1.0 indicates perfect logical entailment
• Critical for abstractive summarization and RAG output validation

A quantitative measure of how well an LLM's output is anchored to a specific retrieved document or knowledge base entry. Essential in Retrieval-Augmented Generation (RAG) systems to prevent model drift.

• Evaluates token-level alignment between output and retrieved context
• Low grounding scores indicate the model is ignoring retrieval and relying on parametric knowledge
• Often paired with citation precision for complete attribution assessment

The frequency at which a language model generates nonsensical or factually incorrect text relative to source material, expressed as a percentage of total generated tokens or sentences.

• Calculated at both entity-level and sentence-level granularity
• A critical KPI in production monitoring dashboards
• Benchmarks like HaluEval and TruthfulQA standardize measurement across models

A composite metric evaluating whether a model can correctly link a generated claim to the specific segment of a source document that supports it. Measured through two sub-metrics:

• Citation Recall: proportion of claims with supporting citations
• Citation Precision: proportion of citations that actually support their claim
• High attribution scores are essential for enterprise compliance and auditability

A human-aligned evaluation metric that breaks long-form generation into atomic facts and verifies each against a trusted knowledge base like Wikipedia. Returns the percentage of supported facts.

• Decomposes complex paragraphs into discrete, verifiable claims
• More granular than sentence-level evaluation
• Used to benchmark models like GPT-4 and Claude for biographical and encyclopedic accuracy

A zero-resource hallucination detection method that samples multiple responses from a black-box LLM and checks for factual inconsistency. Leverages the principle that hallucinated facts are stochastically unstable across samples.

• Requires no external knowledge base or training data
• Uses semantic similarity metrics like BERTScore or NLI between samples
• Effective for detecting entity-level hallucinations in open-domain generation