This prompt is designed for AI engineers and platform teams who need to programmatically verify whether a Retrieval-Augmented Generation (RAG) answer is factually grounded in its source material. The primary job-to-be-done is automated quality assurance: before a generated answer reaches a user, a log, or a downstream system, you need a structured verdict on which claims are supported, which are hallucinated, and which fall into an ambiguous gray zone. The ideal user is someone integrating this into a CI/CD pipeline, a monitoring service, or a batch evaluation job—not a casual user chatting with an LLM. You should have the retrieved context and the final generated answer ready as inputs, and you should expect a machine-readable output (JSON) that can trigger alerts, block publication, or route items for human review.
Prompt
Hallucination Detection Prompt for RAG Answers
When to Use This Prompt
Defines the core job, ideal user, and operational boundaries for the hallucination detection prompt in production RAG pipelines.
Use this prompt when the cost of hallucination is high: customer-facing answers, regulated content, clinical summaries, financial analysis, or any domain where unsupported claims erode trust or create liability. It is specifically built for sentence-level classification, meaning it breaks the answer into atomic claims and judges each against the provided context. This granularity is critical for production monitoring because it allows you to track grounding drift over time, identify specific retrieval gaps, and measure hallucination rates as a precise metric rather than a vague feeling. The prompt includes confidence scores and a severity rubric, making it suitable for automated gating where low-confidence or high-severity hallucinations should stop the pipeline and escalate to a human reviewer.
Do not use this prompt when you lack access to the retrieved context, when the answer is purely creative or opinion-based, or when the source material itself is untrusted. This prompt assumes the provided context is the ground truth; it will not fact-check the context against external reality. It is also not a replacement for a full human review in high-stakes scenarios—treat it as a high-signal filter that reduces the review burden, not as an infallible oracle. If your RAG system retrieves from dozens of documents, pre-filter the context to the most relevant passages before invoking this prompt to avoid diluting the model's attention. Finally, if you need a simple binary hallucination flag without severity or confidence data, a lighter-weight prompt may be cheaper and faster; this prompt is for teams that need diagnostic depth to debug and improve their RAG pipeline over time.
Use Case Fit
Where this prompt works, where it fails, and the operational conditions required before you depend on it in production.
Good Fit: RAG Output Auditing
Use when: You have a RAG pipeline and need to measure groundedness at scale. The prompt excels at sentence-level classification when the retrieved context is clean and the answer is factual. Guardrail: Run this prompt on a sample of production traffic, not every response, to balance cost and coverage.
Bad Fit: Creative or Opinion Tasks
Avoid when: The output is subjective, speculative, or creative. The prompt will flag stylistic choices as hallucinations. Guardrail: Use a different rubric for summarization tone or creative fidelity. This prompt assumes a fact-checking contract.
Required Inputs
Risk: Missing or truncated context breaks the prompt. If the retrieved passages don't contain the evidence, every sentence will be flagged as unsupported. Guardrail: Validate that [RETRIEVED_CONTEXT] is non-empty and relevant before calling the prompt. Log empty-context calls separately.
Operational Risk: Judge Drift
Risk: The LLM judge's severity calibration drifts across model versions or over time. What was 'minor hallucination' last month becomes 'moderate' today. Guardrail: Maintain a golden set of 50 annotated examples and run calibration checks weekly. Pair with a calibration prompt from the Score Calibration pillar.
Operational Risk: Cost at Scale
Risk: Running this prompt on every RAG response is expensive. The prompt is long and requires both the answer and full context. Guardrail: Sample traffic, trigger on low confidence scores from your primary model, or run only on high-risk query categories. Never make this a synchronous blocking step for every user request.
Not a Replacement for Human Review
Risk: Teams treat the prompt's output as ground truth and stop human auditing. The prompt can miss subtle fabrications or over-flag cautious language. Guardrail: Use this prompt as a triage and monitoring signal. Escalate high-severity or ambiguous cases to a human review queue. The prompt is a sensor, not a judge.
Copy-Ready Prompt Template
A reusable prompt template with square-bracket placeholders for classifying each sentence in a RAG answer as grounded, hallucinated, or ambiguous against retrieved context.
This prompt template is designed to be copied directly into your evaluation harness. It instructs the model to act as a strict factual consistency auditor, comparing a generated answer sentence-by-sentence against the provided source documents. The core instruction forces the model to output a structured JSON object containing a classification, confidence score, and reasoning for every sentence, making it suitable for automated monitoring pipelines. Before using this template, ensure you have already retrieved the relevant context chunks and have the full generated answer text ready. This prompt is not a chatbot; it is a programmatic evaluation step.
textYou are a strict factual consistency auditor. Your task is to compare a generated answer against the provided source documents and classify each sentence. ## INPUT [ANSWER] ## CONTEXT [CONTEXT] ## OUTPUT_SCHEMA Return a single JSON object with the key "sentence_analysis", which is an array of objects. Each object must have the following keys: - "sentence_index": integer (0-based index of the sentence in the answer) - "sentence_text": string (the exact text of the sentence) - "classification": string (must be one of "grounded", "hallucinated", "ambiguous") - "confidence": float (a score between 0.0 and 1.0) - "evidence": string (the exact text from the context that supports or contradicts the sentence, or null if none exists) - "reasoning": string (a brief explanation for the classification) ## CLASSIFICATION DEFINITIONS - **grounded**: The sentence is directly supported by the provided context. The information is explicitly stated or is a trivial, non-controversial paraphrase. - **hallucinated**: The sentence contains factual information that contradicts the context or introduces new, unsupported factual claims not found in the context. - **ambiguous**: The sentence is a matter of opinion, a generic statement, a question, or a stylistic flourish that cannot be verified against the context. ## CONSTRAINTS - You MUST analyze every sentence, including introductory or concluding remarks. - If a sentence is a direct quote from the context, it is "grounded". - If a sentence contains multiple clauses with different grounding statuses, classify the entire sentence as "hallucinated" if any part is unsupported or contradictory. - Do not use external knowledge. Base your judgment solely on the provided [CONTEXT]. - Output ONLY the valid JSON object. Do not include any text before or after the JSON.
To adapt this template, replace the [ANSWER] and [CONTEXT] placeholders with your actual data. The [ANSWER] should be the full, unmodified text from your RAG system. The [CONTEXT] should be the concatenated text of all retrieved chunks, with clear separators (e.g., [SOURCE_1] ... [SOURCE_2] ...). For production use, you should validate the output JSON against the defined schema before trusting the results. If the model fails to produce valid JSON, implement a retry mechanism with a simplified prompt or a different model. For high-stakes applications, always route sentences classified as "hallucinated" with high confidence to a human review queue.
Prompt Variables
Required inputs for the Hallucination Detection Prompt. Each variable must be populated before the prompt is sent. Missing or malformed inputs are the most common cause of false negatives in production.
| Placeholder | Purpose | Example | Validation Notes |
|---|---|---|---|
[RAG_ANSWER] | The full generated answer text to evaluate for hallucinations | The patient's blood pressure was 140/90 mmHg, indicating stage 2 hypertension. | Must be non-empty string. If null or whitespace, abort evaluation and return error. Strip leading/trailing whitespace before injection. |
[RETRIEVED_CONTEXT] | The source documents or passages retrieved for answering the query | Document 1: Patient vitals show BP 140/90... Document 2: Stage 2 hypertension is defined as systolic >=140 or diastolic >=90. | Must contain at least one non-empty passage. Validate that context is not identical to answer (prevents circular grounding). Maximum 10,000 tokens to avoid context window overflow. |
[USER_QUERY] | The original question or prompt that triggered the RAG answer | What is the patient's blood pressure and what does it indicate? | Optional but strongly recommended. When provided, enables cross-checking whether hallucinated claims are query-relevant. Null allowed but reduces detection accuracy for off-topic fabrications. |
[CONFIDENCE_THRESHOLD] | Minimum confidence score for classifying a sentence as grounded | 0.7 | Must be float between 0.0 and 1.0. Default 0.7 if not specified. Lower values increase false positives (marking grounded as hallucinated). Higher values increase false negatives (missing real hallucinations). |
[SEVERITY_RUBRIC] | Definitions for hallucination severity levels used in output classification | CRITICAL: fabricated medical advice. MAJOR: wrong numbers. MINOR: imprecise wording. | Must define at least 3 levels. Each level must have a clear, non-overlapping description. Validate that rubric levels are mutually exclusive. Null allowed to use default rubric. |
[OUTPUT_SCHEMA] | Expected JSON structure for the evaluation output | {"sentences": [{"text": "...", "status": "grounded|hallucinated|ambiguous", "confidence": 0.0-1.0, "evidence": "...", "severity": "..."}]} | Must be valid JSON schema. Validate parseability before prompt assembly. If null, use default schema. Schema must include fields for sentence text, status enum, confidence score, and evidence citation. |
[ABSTENTION_RULES] | Conditions under which the evaluator should decline to classify | If context is empty, return status: 'insufficient_evidence' for all sentences. If answer is identical to context, mark all as grounded. | Must be explicit string or null. If null, evaluator attempts classification with available evidence. Validate that rules do not contradict each other. Test with edge cases before production deployment. |
Implementation Harness Notes
How to wire the hallucination detection prompt into a production RAG monitoring pipeline with validation, retries, and human review gates.
This prompt is designed to operate as a post-generation evaluation step in a RAG pipeline, not as a real-time user-facing filter. After your primary LLM generates an answer from retrieved context, pass both the generated answer and the retrieved context chunks into this prompt. The output is a structured JSON report that classifies each sentence as grounded, hallucinated, or ambiguous, with confidence scores and severity labels. This report feeds downstream monitoring dashboards, alerting systems, and human review queues—it is not intended to block the user response in the same request cycle unless your latency budget and risk tolerance explicitly allow synchronous gating.
Wiring the prompt into your application requires a few concrete components. First, implement a sentence splitter that segments the generated answer before calling the prompt—the prompt expects pre-split sentences as input, not raw paragraphs. Use a reliable sentence tokenizer (spaCy, NLTK, or a model-native splitter) and pass the sentence list alongside the retrieved context. Second, wrap the LLM call in a retry wrapper with exponential backoff (3 attempts minimum) because structured JSON outputs from evaluation prompts can occasionally produce parse errors under high load. Third, add a JSON schema validator (using Pydantic, Zod, or JSON Schema) that checks the output structure before ingestion: every sentence must have a classification, confidence, and evidence_span field. Reject and retry any response that fails schema validation. Fourth, log every evaluation result with the original answer, context, and model version to your observability platform for trace analysis and regression testing.
Severity-based routing is the most important architectural decision. Configure your pipeline to route hallucinated sentences with severity: critical or severity: high to a human review queue within your existing ticketing or review system. Low-severity hallucinations (minor imprecision, inconsequential details) can be logged for trend analysis without immediate human intervention. For regulated or high-risk domains, implement a hard gate: if any sentence scores hallucinated with confidence above 0.85, block the answer from reaching the user and surface a fallback response. For lower-risk applications, allow the answer through but attach the hallucination report as metadata for offline analysis. Never silently discard evaluation results—every hallucination detection should produce an observable artifact that your team can query, aggregate, and act on.
Model choice matters. This evaluation prompt works best with models that have strong instruction-following and structured output capabilities. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro are reliable choices for production use. Avoid using smaller or older models (GPT-3.5, Claude Haiku) for this task because they produce more false positives on ambiguous sentences and struggle with the structured output format under complex context. If cost is a concern, run the evaluation prompt on a sampled subset of production traffic (e.g., 10% of responses) rather than degrading accuracy with a cheaper model. For high-throughput systems, consider batching multiple answer-context pairs into a single evaluation call to reduce API overhead, but keep batches small (3-5 pairs) to avoid context-window confusion.
What to avoid: Do not use this prompt as your only safety net. It detects hallucination against provided context, but it cannot detect when the retrieved context itself is wrong, incomplete, or irrelevant. Pair this prompt with retrieval quality metrics (precision@k, recall, context relevance scoring) and upstream factuality checks. Do not treat ambiguous classifications as safe—they indicate the model couldn't determine grounding status, which often signals retrieval gaps or poorly written source material. Route ambiguous sentences to a separate review bucket for retrieval pipeline improvement. Finally, do not skip calibration: run this prompt against a golden dataset of 50-100 human-annotated examples to measure your judge model's precision and recall before relying on it in production. A hallucination detector that misses 30% of fabrications or flags 40% of grounded sentences creates more operational noise than value.
Common Failure Modes
Hallucination detection prompts fail in predictable ways. Here are the most common failure modes and how to guard against them in production.
Judge Hallucinates the Verdict
Risk: The LLM judge fabricates evidence or hallucinates a grounding label, defeating the purpose of the detection prompt. This often happens when the judge is forced to make a binary call on ambiguous claims without enough context. Guardrail: Require the judge to quote the exact source text supporting its verdict. If no quote exists, default to 'ambiguous' or 'unsupported' rather than forcing a grounded label. Add a validator that rejects verdicts without source quotes.
Overly Generous Grounding Classification
Risk: The judge marks claims as 'grounded' when they are only loosely related to the source material, inflating faithfulness scores and letting hallucinations through. This is common when the prompt lacks strict entailment criteria. Guardrail: Define grounding as strict entailment—the source must logically imply the claim, not just share a topic. Include few-shot examples showing borderline cases labeled as 'ambiguous' or 'unsupported' to calibrate the judge's threshold.
Context Window Truncation Silently Drops Evidence
Risk: When the retrieved context plus the answer plus the evaluation prompt exceeds the model's context window, evidence is silently truncated. The judge then marks claims as unsupported because it never saw the supporting passage. Guardrail: Measure token counts before calling the judge. If the combined input exceeds 80% of the context window, split the evaluation into chunks or summarize non-essential context. Log truncation warnings explicitly rather than trusting the model to handle overflow.
Ambiguous Claims Default to the Wrong Label
Risk: Claims that are partially supported or require inference beyond the source text get inconsistently labeled, causing score drift across evaluation runs. The judge may flip between 'grounded' and 'hallucinated' based on subtle phrasing changes. Guardrail: Always include an 'ambiguous' or 'insufficient evidence' category in the rubric. Require the judge to explain its reasoning when selecting this label. Track the rate of ambiguous classifications over time—a sudden drop may indicate the judge is over-committing to binary labels.
Judge Inherits the Same Hallucination Pattern
Risk: If the same model family generates the answer and evaluates it, the judge may share the same blind spots, confidently labeling hallucinations as grounded because both models make the same factual errors. Guardrail: Use a different model family or a stronger model for evaluation than for generation. When this isn't possible, add explicit counterfactual checks: 'Would this claim still be marked grounded if the source said the opposite?' Include adversarial examples in few-shot prompts.
Citation Format Mismatch Breaks Verification
Risk: The judge expects citations in a specific format but the RAG system outputs them differently, causing the judge to miss valid source references and flag grounded claims as unsupported. Guardrail: Normalize citation formats before evaluation. Pass a clear citation schema in the judge prompt showing exactly how source references appear. Add a pre-processing step that extracts and standardizes citations from the answer before the judge sees them.
Evaluation Rubric
Use this rubric to test the Hallucination Detection Prompt before shipping. Each criterion defines a pass standard, a failure signal, and a concrete test method. Run these checks against a curated set of 20-30 RAG answer examples with known grounding labels.
| Criterion | Pass Standard | Failure Signal | Test Method |
|---|---|---|---|
Sentence-level classification accuracy | ≥90% agreement with human-labeled grounded/hallucinated/ambiguous labels across 50 test sentences | System mislabels a clearly hallucinated sentence as grounded or vice versa in >10% of cases | Run prompt against a golden dataset of 50 sentences with human-annotated labels; compute precision/recall per class |
Confidence score calibration | Mean confidence score for correct classifications is ≥0.8; mean confidence for incorrect classifications is ≤0.5 | High confidence (≥0.9) assigned to an incorrect classification or low confidence (≤0.3) assigned to a correct one | Bin predictions by confidence decile; plot accuracy per bin; check for monotonic increase in accuracy with confidence |
Severity rubric consistency | Critical hallucinations are flagged with severity ≥4 in ≥95% of cases where the hallucination introduces a false safety or compliance claim | A fabricated statistic about drug dosage or financial liability receives a severity score of 1 or 2 | Curate 10 examples of critical hallucinations; verify severity scores are ≥4 for all; repeat with 10 minor imprecision examples expecting ≤2 |
Ambiguous sentence handling | Sentences with partial support or mixed evidence are labeled ambiguous ≥80% of the time rather than forced into grounded or hallucinated | Prompt labels a sentence as grounded when only 40% of its claims are supported by the retrieved context | Use 15 sentences with known partial support; measure the rate of ambiguous classification vs. incorrect binary classification |
Multi-document conflict detection | When two retrieved documents contradict each other on a claim, the prompt flags the conflict in the ambiguity reason field | Prompt marks a claim as grounded based on one document while ignoring a direct contradiction in another retrieved passage | Feed RAG outputs with known inter-document conflicts; check that the ambiguity_reason field mentions the conflict explicitly |
Output schema compliance | 100% of responses parse as valid JSON matching the [OUTPUT_SCHEMA] with all required fields present | Missing severity_score field, malformed JSON, or sentence_index values that don't match input sentence order | Validate output with a JSON schema validator; run 100 test calls and require zero parse failures |
Citation-to-claim alignment | Every grounded classification includes a source_passage_id that maps to the exact retrieved passage supporting the claim | A sentence is marked grounded but the source_passage_id points to an irrelevant passage or is null | For 30 grounded classifications, manually verify that the cited passage actually supports the claim; require ≥95% alignment |
Abstention on unverifiable input | When retrieved context is empty or entirely irrelevant, the prompt marks all sentences as ambiguous with reason insufficient_context | Prompt hallucinates a grounded label by treating its own parametric knowledge as retrieved evidence | Test with 10 RAG answers where retrieved context is intentionally empty or off-topic; verify no grounded classifications appear |
Enabling Efficiency, Speed & Accuracy
Intelligent Analysis, Decision & Execution
We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.
Talk to Us
Search across company data
Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.
Useful when people spend too long searching or get different answers from different systems.
Automate internal workflows
Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.
Useful when repetitive work moves across multiple tools and teams.
Add AI to products and internal tools
Build assistants, guided actions, or decision support into the software your team or customers already use.
Useful when AI needs to be part of the product, not a separate tool.
Adapt This Prompt
How to adapt
Use the base prompt with a single model call and lighter validation. Drop the severity rubric and confidence scores initially. Focus on sentence-level binary classification: grounded or hallucinated. Accept raw text output instead of strict JSON.
codeClassify each sentence in [ANSWER] as grounded or hallucinated relative to [CONTEXT].
Watch for
- Model conflating ambiguity with hallucination
- No structured output means harder downstream parsing
- Missing edge cases like partial grounding or implied claims
About the author
Prasad Kumkar
CEO & MD, Inference Systems
Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.
His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.
Partnered with leading AI, data, and software stack.
How We Work
Custom AI workflows for your Business
One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.
01
Review the use case
We understand the task, the users, and where AI can actually help.
Read more02
Pick the right approach
We define what needs search, automation, or product integration.
Read more03
Build the first useful version
We implement the part that proves the value first.
Read more04
Improve from there
We add the checks and visibility needed to keep it useful.
Read moreThe first call is a practical review of your use case and the right next step.
Talk to Us