Hallucination Detection

This technique prompts the model to generate multiple responses to the same query and then cross-checks them for factual and logical consistency. A high degree of variance between answers often signals hallucination.

• Implementation: Use sampling techniques (e.g., temperature > 0) to create n candidate outputs.
• Analysis: Compare the candidates for factual claims, numerical outputs, and logical conclusions.
• Metric: Calculate a semantic similarity score (e.g., using BERTScore or entailment models) between outputs. Low aggregate similarity indicates potential hallucination.

This method requires the model to cite specific source passages for any factual claim it makes. The cited text is then retrieved and compared to the generated claim for faithfulness.

• Process: In a Retrieval-Augmented Generation (RAG) pipeline, force the model to output [Citation: X] tags.
• Verification: For each claim, retrieve the source text indicated by the citation ID.
• Evaluation: Use a Natural Language Inference (NLI) model to judge if the claim is entailed by, contradicts, or is neutral to the source. Non-entailed claims are flagged.

This statistical approach flags sentences or tokens that are highly surprising to the model itself, indicated by a sharp spike in local perplexity. While not definitive for factual errors, it effectively detects nonsensical or out-of-distribution phrasing.

• Mechanism: Calculate the perplexity (PPL) for each token in the generated sequence using the same model that produced it.
• Thresholding: Tokens or spans with PPL significantly above the sequence's baseline are potential hallucinations.
• Use Case: Particularly effective for detecting intrinsic hallucinations (contradictions within the generated text).

Specialized natural language inference models are used as external verifiers. These models are trained to detect logical relationships between statements.

• Workflow: Pass the source context (or a knowledge base retrieval) and the model's generated claim as a premise-hypothesis pair to an NLI model (e.g., DeBERTa, RoBERTa).
• Classification: The verifier outputs a label: ENTAILMENT, CONTRADICTION, or NEUTRAL.
• Action: CONTRADICTION labels are clear hallucinations. NEUTRAL claims may require further verification, as they are unsupported.

These are automated, reference-free metrics designed to quantify the factual alignment between a generated summary or answer and its source document.

• BERTScore: Computes precision, recall, and F1 based on token-level similarity using contextual embeddings from BERT. It assesses if key entities and relations from the source are preserved.
• QAFactEval: A more robust metric that operates by:
  1. Generating question-answer pairs from the source.
  2. Answering those same questions from the generated text.
  3. Comparing the answers. Low scores indicate missing or altered facts.

The most reliable but costly method involves human experts evaluating model outputs against established ground truth or verifiable sources. This creates labeled datasets for training automated detectors.

• Process: Domain experts annotate model outputs for categories like Factual Correctness, Completeness, and Faithfulness.
• Outcome: Produces gold-standard evaluation sets used to benchmark automated techniques.
• Scalability: This data is crucial for fine-tuning smaller critic models or reward models that can approximate human judgment at scale for specific domains.

A confidence score is a numerical measure, typically a probability between 0 and 1, that a machine learning model assigns to its prediction to indicate its certainty. In hallucination detection, a low confidence score on a generated fact can be a primary signal for potential fabrication.

• Key Role: Serves as a first-pass filter for identifying low-reliability outputs that warrant deeper verification.
• Calibration: A well-calibrated model's confidence score should match its empirical accuracy; miscalibration can lead to missed hallucinations or false alarms.
• Application: Used in retrieval-augmented generation (RAG) systems to decide when to abstain from answering or to trigger a fact-checking subroutine.

Calibration error quantifies the discrepancy between a model's predicted probabilities and the true empirical frequencies of outcomes. A model with high calibration error may be overconfident in its hallucinations or underconfident in its correct outputs.

• Measurement: Common metrics include Expected Calibration Error (ECE) and Maximum Calibration Error (MCE), which bin predictions by confidence and compare to accuracy within each bin.
• Impact on Detection: Poor calibration directly undermines the utility of confidence scores for reliable hallucination flagging.
• Mitigation: Techniques like temperature scaling and Platt scaling are used post-training to improve a model's calibration.

Anomaly detection is the broader process of identifying rare items, events, or observations in data that deviate significantly from the majority or an expected pattern. Hallucination detection is a domain-specific application of anomaly detection focused on nonsensical or unfaithful model outputs.

• Technical Foundation: Leverages statistical tests, distance-based methods (e.g., k-NN), density-based methods (e.g., Local Outlier Factor), or reconstruction-based autoencoders.
• Feature Space: For hallucinations, anomalies may be detected in the embedding space of generated text, in the distribution of token probabilities, or in the contradiction between a claim and a knowledge source.
• Unsupervised Nature: Often operates without labeled examples of "hallucinations," identifying outliers based on intrinsic data properties.

Precision (fraction of flagged items that are actual hallucinations) and Recall (fraction of all hallucinations that are successfully flagged) are the fundamental metrics for evaluating any binary hallucination detection classifier.

• Trade-off: Increasing detection sensitivity (recall) often increases false positives, lowering precision. The optimal balance depends on the application's cost of missing a hallucination vs. the cost of unnecessary verification.
• F1 Score: The harmonic mean of precision and recall (F1 Score) provides a single metric to compare detectors.
• Context: In safety-critical applications (e.g., medical advice), high recall is paramount. In content generation for drafting, high precision may be preferred to avoid interrupting workflow.

In the statistical framework of hallucination detection, a Type I error (false positive) occurs when correct model output is incorrectly flagged as a hallucination. A Type II error (false negative) occurs when an actual hallucination is missed.

• Error Analysis: Understanding the prevalence and cause of each error type is essential for refining detection systems. Type I errors can degrade user trust, while Type II errors can propagate misinformation.
• Cost Asymmetry: The cost of a Type II error is often significantly higher, driving the design of detection systems toward higher sensitivity, albeit with managed false positive rates.
• Root Cause: Type I errors may stem from overly strict factual verification or poor source retrieval. Type II errors often arise from subtle contradictions or plausible-sounding fabrications.

Retrieval-Augmented Generation is an architecture designed to mitigate hallucinations by grounding a large language model's responses in retrieved evidence from an external knowledge source. Hallucination detection in a RAG system involves verifying the faithfulness of the generated text to the provided context.

• Detection Mechanism: Techniques include answer attribution, which requires the model to cite specific source passages, and contradiction checking between the generation and the retrieved context using a natural language inference (NLI) model.
• Faithfulness Metrics: Metrics like BERTScore or QAFactEval can automatically measure the factual overlap between a generated answer and its source context.
• Architectural Role: RAG provides the necessary source material against which hallucinations can be detected, making detection feasible and interpretable.