Faithfulness Metric

The metric must decompose generated text into atomic claims and verify each against the source document using a Natural Language Inference (NLI) model. A claim is faithful only if it is entailed by the premise text.

• Contradiction or neutral labels indicate hallucination
• Score is typically the ratio of entailed claims to total claims
• Example: 'The cat sat on the mat' entails 'A feline was on a floor covering'

The score must remain stable regardless of how finely the text is segmented. Whether evaluating at the sentence level, clause level, or proposition level, the aggregate faithfulness score should converge.

• Prevents gaming by manipulating sentence boundaries
• Requires atomic fact decomposition for true precision
• Proposition-level scoring is the gold standard for detecting subtle hallucinations

A faithful metric enforces a closed-world assumption: the source document is the sole ground truth. Plausible but unsupported statements are penalized as harshly as obvious falsehoods.

• Extrinsic knowledge is treated as hallucination
• Prevents 'helpful' additions that introduce factual risk
• Critical for regulated domains like legal and medical summarization

A single hallucinated claim must not be masked by a high volume of faithful ones. The metric should use minimum or harmonic mean functions rather than simple arithmetic averaging.

• A summary with one fabricated drug dosage is dangerous, even if 99% correct
• Harmonic mean penalizes low scores more severely
• Enables strict pass/fail thresholds in automated pipelines

The metric must be calibrated so that a score of 1.0 guarantees zero hallucinations, and a score of 0.0 indicates complete fabrication. Intermediate values must correlate monotonically with human judgments of factual accuracy.

• Requires benchmarking against human-annotated datasets like FRANK or XSum-Faithful
• Enables setting reliable operational thresholds for blocking content

While primarily measuring factual accuracy, a robust metric should also flag critical omissions where the source contains material information absent from the summary. This is often handled by a complementary recall or coverage score.

• Faithfulness alone does not guarantee completeness
• A summary that says nothing is perfectly faithful but useless
• Combined with a coverage metric for a full quality picture

A system that tracks the gradual shift in the meaning or contextual relevance of generated content over time. In long-form generation, models can wander away from the original source material, introducing subtle hallucinations.

• Compares rolling windows of output against the source document
• Detects topic divergence before it becomes overt hallucination
• Essential for long-document summarization tasks