Factual Consistency Scoring

At its core, the metric relies on an NLI model to assess the directional relationship between a source text (premise) and a generated claim (hypothesis).

• Entailment: The claim is fully supported by the source.
• Contradiction: The claim directly opposes the source.
• Neutral: The claim contains information not discussed in the source.

This framework provides a structured, token-level judgment of logical alignment rather than simple n-gram overlap.

A robust scoring pipeline first atomizes a long-form summary into discrete, self-contained atomic claims. Each atomic fact is then independently verified against the source documents.

• Isolates errors: Prevents a single hallucination from invalidating an otherwise accurate paragraph.
• Enables precision: Calculates the exact percentage of generated claims that are factually grounded.
• Supports granular debugging: Identifies the specific type of failure (e.g., entity swap, date error).

The primary function is to penalize hallucinations, not just reward relevance. The scoring function applies a heavy negative weight to any claim classified as a contradiction.

• Binary strictness: A single contradicted fact can severely degrade the overall score.
• Source fidelity: Ensures the summary does not introduce external world knowledge that conflicts with the provided context.
• Safety mechanism: Acts as a critical filter for high-stakes domains like medical or legal synthesis where fabrication is unacceptable.

The final consistency score is typically reported as an F1-score, balancing precision and recall of supported claims.

• Precision: The fraction of claims in the summary that are entailed by the source.
• Recall: The fraction of source-relevant claims that were correctly included.
• Harmonic mean: Provides a single, balanced metric that penalizes both verbose hallucination and incomplete summarization.

Modern NLI-based scoring models are trained on diverse textual entailment datasets, allowing them to generalize to new domains without task-specific fine-tuning.

• Domain agnostic: Works on financial reports, clinical notes, or legal briefs without retraining.
• Synthetic data resilience: Can validate summaries generated from previously unseen document structures.
• Rapid deployment: Enables immediate quality assurance for new retrieval-augmented generation (RAG) pipelines.

The ultimate validation of a factual consistency metric is its statistical correlation with expert human evaluators. Top-performing metrics achieve high Pearson and Spearman correlation scores on benchmark datasets like SummaC and TRUE.

• Gold standard alignment: Ensures the automated score is a reliable proxy for manual review.
• Reduces evaluation cost: Minimizes the need for slow, expensive human annotation in the development loop.
• Benchmarking rigor: Allows for objective comparison of different summarization models' hallucination rates.