NLI-Based Evaluation

At its core, NLI-based evaluation classifies the relationship between a premise (the grounding document) and a hypothesis (the generated claim) into three categories:

• Entailment: The hypothesis is logically supported by the premise.
• Contradiction: The hypothesis directly opposes the premise.
• Neutral: The hypothesis may be true but is not verifiable from the premise. This framework, derived from the Stanford NLI (SNLI) and Multi-Genre NLI (MultiNLI) benchmarks, provides a rigorous, interpretable signal for factual grounding.

Standard NLI operates at the sentence level, but modern evaluation requires finer granularity. Factual Consistency NLI decomposes summaries into atomic claims and checks each against the source:

• A claim is a minimal, self-contained factual statement.
• Each claim is paired with the full source document as the premise.
• The model predicts an entailment probability for each claim. The final Faithfulness Score is the proportion of claims classified as entailed, providing a direct, interpretable metric for hallucination severity.

A dedicated NLI model, often a fine-tuned variant of RoBERTa or DeBERTa, acts as the evaluator. This model is distinct from the generator being tested. Key training datasets include:

• DocNLI: A large-scale dataset for document-level NLI.
• SummaC: Specifically designed for summary consistency checking.
• ANLI: Adversarial NLI examples to improve robustness. The evaluator's accuracy is critical; a weak NLI model will produce unreliable faithfulness scores, making evaluator calibration a prerequisite for deployment.

Modern large language models can perform NLI-based evaluation without task-specific fine-tuning. Zero-shot NLI prompts a general-purpose LLM to classify the premise-hypothesis pair:

• The prompt defines the three classes and requests a structured judgment.
• Chain-of-Thought (CoT) prompting can improve accuracy by asking the model to explain its reasoning before classifying.
• This approach eliminates the need for a separate, fine-tuned evaluator model but introduces higher latency and cost per evaluation.

NLI-based evaluation has known failure modes that must be accounted for in production pipelines:

• Extrinsic Hallucinations: Claims that are true in the world but not in the source document are often misclassified as neutral rather than contradictory.
• Paraphrase Blindness: High semantic overlap can cause an NLI model to predict entailment even when a subtle factual error exists.
• Temporal Reasoning: NLI models struggle with time-sensitive facts, failing to detect contradictions involving dates or sequences. Mitigation involves combining NLI scores with Question Answering-based verification for a more robust composite metric.

In Retrieval-Augmented Generation systems, NLI evaluation serves as a real-time guardrail:

• The retrieved context acts as the premise.
• The generated response is decomposed into hypotheses.
• Responses with entailment scores below a threshold are flagged for regeneration or human review. This creates a factual consistency feedback loop, where low-scoring outputs trigger re-retrieval with reformulated queries, directly reducing the hallucination rate in production.