Hallucination Detection

This technique quantifies the model's internal certainty about its own outputs. Low-confidence scores or high predictive entropy often signal potential hallucinations.

• Perplexity: Measures how surprised the model is by its own generated token sequence. Abnormally high perplexity can indicate nonsensical output.
• Token Probabilities: Analyzing the probability distribution over the vocabulary for each generated token. A flat or highly uncertain distribution suggests the model is 'guessing'.
• Monte Carlo Dropout: A Bayesian approximation method that runs multiple forward passes with dropout enabled at inference to estimate predictive uncertainty.

This method grounds model outputs by cross-referencing them against a trusted knowledge source, typically a vector database or search index.

• Embedding Similarity Check: Encodes the generated claim and relevant source passages into vector embeddings (e.g., using a model like text-embedding-3-small). A low cosine similarity score indicates the output is semantically distant from its supposed source.
• Claim Decomposition: Breaks a complex generated statement into individual atomic claims, each of which is independently verified against retrieved evidence.
• Citation Verification: Checks if citations provided by the model are accurate and that the referenced text actually supports the generated claim.

Detects hallucinations by identifying logical inconsistencies within a single output or across multiple turns of a conversation.

• NLI (Natural Language Inference) Models: Uses a pre-trained model (e.g., DeBERTa for MNLI) to check if different parts of the generated text entail, contradict, or are neutral to each other. A contradiction label signals a hallucination.
• Multi-Hop Consistency Checks: For long-form generation, verifies that facts stated earlier in the text are not contradicted later.
• Cross-Model Consistency: Generates the same answer to a query using multiple models or sampling techniques and flags outputs where the core factual claims diverge significantly.

Leverages structured knowledge bases to perform deterministic fact-checking against established entities and relationships.

• Entity Linking & Disambiguation: Identifies named entities (people, places, organizations) in the generated text and links them to canonical entries in a knowledge graph (e.g., Wikidata, an enterprise KG).
• Relationship Validation: Queries the knowledge graph to verify if the predicted relationship between two entities (e.g., 'invented by', 'located in') actually exists.
• Temporal Consistency Check: Validates that dates and event sequences mentioned in the output are chronologically possible according to the knowledge graph.

Uses carefully designed follow-up prompts to force the model to reveal the lack of grounding for a hallucinated claim.

• Source Request: After an answer is generated, prompt the model with: 'Quote the exact sentences from the provided context that support your answer.' An inability to provide a direct quote is a strong indicator.
• Confidence Elicitation: Directly ask the model to rate its confidence on a scale and provide reasoning. Hallucinations are often accompanied by overconfident but vague justifications.
• Alternative Generation: Ask the model to generate alternative answers or viewpoints. A hallucinated 'fact' may be presented as the only possible answer, while a grounded fact allows for nuanced alternatives.

Applies general anomaly detection algorithms to model outputs, treating hallucinations as statistical outliers.

• n-gram Overlap (ROUGE, BLEU): While primarily evaluation metrics, unusually low overlap with relevant source text can indicate the model has diverged into fabrication.
• Stylometric Analysis: Detects shifts in writing style, complexity, or vocabulary that differ from the model's typical grounded outputs, which can be a marker of 'confabulation'.
• Ensemble Disagreement: Uses a committee of diverse models (e.g., different architectures, sizes) to answer the same query. Outputs where the ensemble shows high disagreement are flagged for potential hallucination.

The model is prompted to break its own answer into discrete, verifiable claims and then assess each one. This leverages the model's internal knowledge to perform a form of self-critique.

• Process: Use a follow-up prompt: "List all factual claims in the above answer. For each claim, state if it is true, false, or unverifiable."
• Entailment Models: For automated pipelines, use a specialized Natural Language Inference (NLI) model (e.g., DeBERTa fine-tuned on MNLI) to check if the claim is entailed by the source context.
• Output: A confidence score based on the percentage of verified claims. The presence of any 'false' claims triggers a hallucination alert.

Mitigates single-model bias by using multiple LLMs or specialized classifiers to validate the same output. Disagreement between models signals potential issues.

• Diverse Model Querying: Generate an answer with a primary model (e.g., GPT-4), then ask a different model (e.g., Claude 3, Gemini) to fact-check it against provided sources.
• Specialized Detectors: Employ models fine-tuned specifically for hallucination detection, such as Google's TRUE model or Meta's Search-Augmented Factuality Evaluator (SAFE).
• Voting System: The final detection result is determined by a majority vote or a weighted confidence score from the ensemble.

Validates generated statements against a structured enterprise knowledge graph. This provides a deterministic source of truth for entities and their relationships.

• Process: Extract entities and relations from the generated text using a Named Entity Recognition (NER) and Relation Extraction pipeline.
• Query: Formulate a graph query (e.g., Cypher for Neo4j) to check if the extracted relationship exists between the entities in the knowledge base.
• Result: Statements describing relationships not present in the graph are flagged. This is highly effective for detecting hallucinations about organizational facts, product specs, or process rules.

Leverages the model's own token-level probability scores to identify low-confidence, potentially hallucinated segments. High perplexity indicates the model is "surprised" by its own continuation.

• Mechanism: Monitor the per-token probability or perplexity of the generated sequence. A sudden spike in perplexity often corresponds to nonsensical or factually dubious text.
• Implementation: Access the model's logits during generation. Calculate the perplexity for sliding windows of the output text.
• Use Case: Particularly useful for detecting intrinsic hallucinations—contradictions within the generated text itself—where the model's confidence becomes inconsistent.

Output validation is the systematic process of verifying that data generated by an AI system meets predefined criteria for correctness, format, safety, and business rule adherence. It is the umbrella category under which hallucination detection operates.

• Purpose: To act as a quality gate before an output is accepted or delivered.
• Methods: Can include schema checks, rule-based logic, semantic analysis, and statistical confidence scoring.
• Example: Validating that a JSON response from an agent contains all required fields with the correct data types before it's passed to a downstream API.

A guardrail is a software control designed to constrain AI behavior and prevent outputs that are unsafe, off-topic, biased, or otherwise violate policy. They are proactive filters, whereas hallucination detection is often a diagnostic check.

• Function: Intercepts and modifies or blocks non-compliant outputs.
• Implementation: Can be rule-based (keyword blocklists) or model-based (classifiers for toxicity).
• Example: A guardrail preventing a customer service chatbot from making medical diagnoses or sharing internal API keys.

A confidence threshold is a predefined cutoff value (e.g., 0.85) for a model's output probability or score. Outputs with confidence below this threshold are rejected, flagged for review, or trigger corrective actions like hallucination detection routines.

• Role: Provides a statistical gate for uncertainty management.
• Application: Used in conjunction with conformal prediction to provide rigorous uncertainty quantification.
• Example: An LLM-based summarization agent discards any summary where its internal confidence score is below 90%, as low confidence correlates with potential factual errors.

Citation verification is the process of checking the accuracy and support of references provided by an AI system. It is a direct method for detecting hallucinations in retrieval-augmented generation (RAG) systems by ensuring claims are grounded in source material.

• Process: Involves cross-referencing the generated statement with the cited source document.
• Metrics: Checks for correct attribution, contextual support, and absence of contradictory information.
• Example: A legal research agent asserts a case law precedent; verification involves retrieving the cited case and confirming the assertion matches the ruling's text.

An embedding similarity check is a validation technique that measures semantic relatedness between two pieces of text using their vector representations. It's used to detect hallucinations by comparing a generated output to its source context.

• Mechanism: Encodes both the claim and the source into high-dimensional vectors (embeddings) and calculates their cosine similarity.
• Use Case: In a RAG pipeline, a low similarity score between an answer chunk and its retrieved context suggests the model may have invented information.
• Tooling: Commonly implemented using models like OpenAI's text-embedding-ada-002 or open-source alternatives from sentence-transformers.

Rule-based validation is a deterministic verification method where outputs are checked against a set of explicit, human-defined logical rules. It provides a fast, interpretable layer of validation complementary to statistical hallucination detection.

• Characteristics: Rules are if-then statements enforcing format, logic, or business constraints.
• Strengths: Provides absolute guarantees for well-defined properties (e.g., "a date must be in YYYY-MM-DD format").
• Example: Validating that an AI-generated SQL query contains a WHERE clause for queries on tables with over 1 million rows to prevent accidental full-table scans.