Factual Entailment Ratio

FER is computed using a Natural Language Inference (NLI) model. This model classifies the relationship between a premise (the source text) and a hypothesis (the AI's claim) into three categories: entailment, contradiction, or neutral. The ratio is calculated as the probability mass assigned to the entailment class.

• Entailment: The source text logically implies the claim is true.
• Contradiction: The source text logically implies the claim is false.
• Neutral: The source text provides no information to determine the claim's truth.

A single AI-generated sentence often contains multiple atomic facts. A robust FER system first performs claim decomposition to break down complex statements into individual verifiable units. Each atomic claim is then independently scored against the cited source.

• Atomic Claim: "Acme Corp was founded in 2010 by Jane Doe."
• Decomposes to: ["Acme Corp was founded in 2010", "Acme Corp was founded by Jane Doe"]
• Result: An aggregate FER is calculated as the harmonic mean of all atomic claim scores, preventing a single verified fact from masking an unsupported one.

FER is robust to lexical variation. It does not require exact keyword overlap. The underlying NLI model is trained to recognize directional logical relationships and paraphrastic entailment.

• Example: A source stating "The patient presented with a pyrexia of 102°F" entails the claim "The patient had a high fever."
• Contrapositive: A source stating "All access was granted after 9 AM" contradicts the claim "User X accessed the system at 8:55 AM."
• This capability is essential for validating AI summaries that naturally rephrase source material.

The precision of FER is directly tied to the granularity of the evidence span. Scoring an entire document against a single claim introduces noise. High-integrity systems use a passage retrieval step before NLI.

• Document-Level FER: Low accuracy, high recall. Prone to false positives.
• Passage-Level FER: The standard approach. A retrieval model first identifies the most relevant 3-5 sentences from the source.
• Sentence-Level FER: The highest precision. The claim is scored only against the single most relevant sentence, providing a definitive alignment signal.

A raw probability is converted into a binary or categorical decision using calibrated thresholds. These thresholds are tuned based on the risk tolerance of the application.

• High-Recall Threshold (e.g., >0.5): Flags any claim with a majority entailment probability as "Supported." Used for initial filtering.
• High-Precision Threshold (e.g., >0.9): Only claims with a very high probability are marked "Verified." Used for final output in regulated industries.
• Contradiction Flag (<0.1): A very low entailment score, combined with a high contradiction score, actively marks a claim as "Refuted" by the source, a critical signal for hallucination detection.

FER is a direct, post-hoc measurement of extrinsic hallucination. While a Hallucination Risk Index predicts the likelihood of error before generation, FER measures the actual factual grounding after generation.

• Low FER + High Confidence: A classic signature of a hallucination where the model is confidently wrong.
• High FER: Provides a verifiable anchor, serving as a positive signal for the Verifiable Claim Ratio.
• FER Trend Analysis: Tracking FER over time for a specific model or pipeline provides an empirical measure of grounding degradation or improvement.

A composite metric that quantifies the degree of semantic and factual correspondence between a specific AI-generated statement and the content of its cited source. While Factual Entailment Ratio measures logical support, Alignment Score captures broader dimensions:

• Semantic overlap: Cosine similarity between claim and source embeddings
• Factual consistency: Whether numerical values, dates, and named entities match exactly
• Context preservation: Whether the source's original meaning is maintained or distorted

A high Alignment Score with a low Entailment Ratio may indicate the source is topically relevant but doesn't actually prove the claim.

A measurement of the semantic distance between the content of a cited source and the AI's generated text, flagging potential misinterpretations or unsupported extrapolations. This metric serves as the inverse counterpart to Factual Entailment Ratio:

• Low divergence + high entailment: Strong, faithful citation
• Low divergence + low entailment: Source is relevant but doesn't logically support the claim
• High divergence + high entailment: Unusual pattern suggesting possible citation fabrication
• High divergence + low entailment: Clear misalignment requiring rejection

Divergence is typically calculated using cross-encoders that directly compare claim-source pairs rather than relying solely on embedding similarity.

A predictive score estimating the likelihood that a generated statement is a hallucination, calculated by analyzing the absence of supporting citations and internal model uncertainty signals. Factual Entailment Ratio serves as a critical input feature:

• Low entailment ratio across all cited sources strongly elevates hallucination risk
• Combined with token-level uncertainty from the model's output distribution
• Integrated with attention pattern analysis to detect when the model ignored its context
• Used alongside semantic consistency checks across multiple generated outputs

The index enables real-time flagging of high-risk claims before they reach end users.

A verification technique that checks for agreement among multiple independent, high-quality sources to confirm a claim, increasing confidence through corroboration. Factual Entailment Ratio is calculated per source, then aggregated:

• Each source independently evaluated for entailment
• Consensus threshold: Minimum number of agreeing sources required
• Source independence check: Ensures sources aren't citing each other circularly
• Weighted aggregation: Higher-quality sources contribute more to the consensus score

This approach mirrors academic peer review by requiring multiple independent verifications before accepting a claim as grounded.

The proportion of factual statements in an AI-generated text that can be successfully verified against a trusted corpus, serving as a key indicator of overall output reliability. Factual Entailment Ratio is the per-claim building block:

• Formula: (Number of claims with entailment ratio > threshold) / (Total factual claims)
• Excludes opinions: Only objective, verifiable statements are counted
• Threshold calibration: Domain-specific; scientific claims require higher entailment than general knowledge
• Trending metric: Tracked over time to detect model drift or retrieval degradation

A Verifiable Claim Ratio below 0.85 typically triggers automated review in production systems.

A measure of the completeness and logical validity of the path from an AI's output claim back through its citations to the foundational, verifiable data. Factual Entailment Ratio evaluates each link in this chain:

• Link-level entailment: Each citation-to-claim relationship scored independently
• Chain completeness: Whether all intermediate sources are accessible and valid
• Transitive validity: If Source A entails Claim B, and Source C cites A, does C faithfully represent A?
• Break detection: Identifies where the evidence chain weakens or breaks entirely

This metric prevents the common failure mode where an AI correctly cites a source that itself misrepresents its own references.