Hallucination Detection

This technique verifies an LLM's output against a trusted knowledge source or the provided context window. It is fundamental to Retrieval-Augmented Generation (RAG) systems.

• Process: Extracts claims or entities from the generated text and queries a database or source documents for verification.
• Metrics: Uses precision and recall to measure the system's ability to identify supported vs. unsupported statements.
• Example: A model claims "The Eiffel Tower is in London." The system checks this against a known geographic database and flags it as a hallucination.

This method leverages the model's own reasoning to detect inconsistencies. The model is prompted to critique or verify its initial output.

• Techniques: Include Chain-of-Verification (CoVe), where the model plans, answers, generates verification questions, and then revises its answer.
• Process: The model is asked: "Are there any factual inaccuracies in the following text?" or "Is every statement in this paragraph supported by the provided context?"
• Benefit: Does not always require an external database, using the model's parametric knowledge as a consistency check.

Multiple specialized machine learning classifiers are applied in sequence or parallel to an LLM's output to flag potential hallucinations.

• Typical Chain: A factuality classifier (trained to distinguish supported/unsupported claims) may follow a toxicity classifier and a PII detector.
• Ensemble Approach: Combines scores from different classifiers (e.g., for contradiction, entailment, semantic similarity to source) into a final risk score.
• Implementation: Often deployed as a post-processing guardrail layer in the inference pipeline before the response is sent to the user.

This technique analyzes the model's internal token probabilities and confidence scores to identify low-certainty generations that may be hallucinations.

• Perplexity: High perplexity (model's surprise at its own output) can indicate nonsensical or out-of-distribution text.
• Token Probability Variance: Erratic shifts in probability distributions across generated tokens can signal a lack of grounding.
• Limitation: A model can be highly confident in its hallucinations, so this is often used in conjunction with other methods.

For high-stakes applications, human reviewers assess outputs flagged as high-risk by automated systems or sampled randomly for quality assurance.

• Workflow: Automated systems assign a hallucination risk score; outputs above a threshold are queued for human verification.
• Role: Humans provide definitive labels, which are then used to retrain detection classifiers and improve automated systems.
• Use Case: Critical in domains like medical informatics, legal reasoning, and financial reporting, where absolute accuracy is paramount.

Proactive, systematic testing where dedicated teams craft inputs designed to trigger hallucinations, probing the model's boundaries and failure modes.

• Goal: To discover vulnerabilities before deployment, informing the development of more robust detection and prevention systems.
• Methods: Include asking for details on obscure topics, requesting contradictory information, or using prompt injection to confuse the model's grounding.
• Outcome: Findings are used to create safety benchmarks and harden models against specific attack vectors.

The automated verification of generated statements against trusted, up-to-date knowledge sources or databases to assess factual accuracy. Unlike general hallucination detection, fact-checking is a targeted process that often involves:

• Retrieval-Augmented Generation (RAG): Using retrieved documents as the ground truth for verification.
• Claim Decomposition: Breaking a complex model output into individual atomic claims for validation.
• Source Attribution: Requiring the model to cite its supporting evidence, enabling traceability. This is critical for applications in finance, healthcare, and legal domains where factual precision is non-negotiable.

The process of checking whether an LLM's output is substantiated by and correctly references the source material or context provided to it. This is a cornerstone of Retrieval-Augmented Generation (RAG) systems. Key mechanisms include:

• Answerability Detection: Determining if a query can be answered from the provided context at all.
• Attribution Scoring: Quantifying how well each part of the generated answer aligns with specific snippets of source text.
• Contradiction Detection: Identifying if the generated statement directly contradicts the provided grounding documents. Failure in grounding verification is a primary cause of hallucinations in enterprise RAG applications.

Software layers and runtime systems applied to LLM inputs and outputs to enforce safety, security, and compliance policies, acting as a proactive filter for hallucinations and other undesirable outputs. They function by:

• Input/Output Scanning: Using specialized classifiers to detect policy violations before or after generation.
• Constrained Decoding: Limiting the model's vocabulary during inference to prevent certain tokens or phrases.
• Schema Enforcement: Forcing outputs to adhere to a predefined JSON or grammatical structure, reducing open-ended nonsense. Frameworks like NVIDIA NeMo Guardrails and Guardrails AI provide programmable interfaces for implementing these controls.

A training and self-improvement methodology developed by Anthropic where an AI model critiques and revises its own outputs according to a set of high-level principles or rules (a 'constitution'). This reduces harmful or untruthful outputs by:

• Self-Critique: The model generates a critique of its initial response based on constitutional principles.
• Self-Revision: The model then rewrites its response to address the critique.
• Reinforcement Learning: This process creates preference data for fine-tuning, baking safety and truthfulness directly into the model's weights rather than relying solely on post-hoc detection. It addresses the root cause of some hallucinations during the model's reasoning process.

An ensemble moderation technique where multiple specialized machine learning classifiers are applied sequentially or in parallel to validate an LLM output. This is a common architectural pattern for comprehensive safety screening. A chain might include:

• Toxicity Classifier: Detects offensive or harmful language.
• PII Detector: Identifies unmasked personally identifiable information.
• Hallucination Detector: Flags potentially factually incorrect statements.
• Bias Detector: Scores for unfair demographic skews. The output of each classifier informs a final moderation decision, allowing for granular, policy-driven actions (e.g., block, rewrite, flag for human review).

The proactive, adversarial testing of an LLM system by dedicated teams who attempt to discover vulnerabilities, safety failures, or harmful outputs—including hallucinations—through systematic probing. This human-in-the-loop process involves:

• Adversarial Prompt Engineering: Crafting inputs designed to elicit factually incorrect, contradictory, or nonsensical responses.
• Scenario Testing: Simulating edge-case user interactions and high-stakes domains.
• Vulnerability Cataloging: Documenting successful 'jailbreaks' or hallucination triggers to improve automated detection systems and model training. Red teaming provides a critical, exploratory complement to automated hallucination detection, uncovering novel failure modes.