Domain-Specific Hallucination Detection Prompt for Medical Outputs

This prompt is designed for healthcare AI teams who need to detect clinically significant hallucinations before medical outputs reach patients or clinicians. Use it when your system generates medical summaries, clinical notes, discharge instructions, or patient-facing explanations from source evidence such as EHR data, clinical guidelines, or medical literature. The prompt flags fabricated findings, incorrect dosages, unsupported diagnoses, missing contraindications, and temporal inconsistencies that could cause patient harm. It is built for a guardrail position in your pipeline: after generation, before human review or release.

This is not a general-purpose hallucination detector. It assumes clinical terminology, structured source evidence, and a mandatory human-in-the-loop review step for any flagged output. The prompt expects a specific input structure: a generated medical text and a set of source evidence passages. It returns a structured JSON report with flagged spans, severity classifications, and grounding assessments. Do not use this prompt for non-medical content, for real-time streaming outputs without adaptation, or in any workflow that bypasses human review of flagged items. The prompt's value is in its clinical specificity—it looks for medication name mismatches, lab value fabrications, and temporal inconsistencies that a general hallucination detector would miss.

Before integrating this prompt, ensure your pipeline can supply clean source evidence and handle the structured JSON output. You will need to map the prompt's severity classifications to your own escalation logic. Start by running the prompt against a labeled dataset of known hallucinations and valid outputs to calibrate thresholds for your specific use case. The next section provides the copy-ready template with all required placeholders.

This prompt template is designed to be dropped directly into your medical output validation pipeline. It instructs the model to act as a clinical hallucination auditor, comparing a generated medical text against provided source evidence. The prompt is structured to produce a machine-readable JSON output that can be parsed by downstream guardrails, logging systems, or human review queues. Before using it, you must replace the square-bracket placeholders with your specific clinical context, the generated output under review, and the source evidence that should support it.

textCopy
You are a clinical hallucination auditor. Your task is to compare a generated medical output against provided source evidence and identify clinically significant hallucinations. A clinically significant hallucination is a statement that could affect patient care, clinical decision-making, or medical understanding if acted upon.

## INPUT
Generated Medical Output:
[GENERATED_OUTPUT]

Source Evidence:
[SOURCE_EVIDENCE]

Clinical Context:
[CLINICAL_CONTEXT]

## INSTRUCTIONS
1. Extract every clinical claim from the Generated Medical Output. A clinical claim includes findings, diagnoses, medications, dosages, procedures, lab results, patient demographics, allergies, contraindications, and care instructions.
2. For each claim, search the Source Evidence for direct support. A claim is supported if the evidence explicitly states it or if it can be directly and unambiguously inferred.
3. Classify each unsupported claim into one of the following hallucination types:
   - `fabricated_finding`: A clinical observation or result not present in the source evidence.
   - `incorrect_dosage`: A medication dosage, frequency, or route that contradicts or is absent from the source evidence.
   - `unsupported_diagnosis`: A diagnosis stated without evidence support.
   - `missing_contraindication`: A failure to mention a documented contraindication, allergy, or interaction when recommending a treatment.
   - `demographic_error`: Incorrect patient age, sex, or other identifying clinical factor.
   - `temporal_error`: A date, time, or sequence of events that contradicts the source evidence.
   - `omitted_material_fact`: A clinically relevant fact from the source evidence that is absent from the generated output, where omission could lead to harm.
4. Assign a severity to each hallucination:
   - `critical`: Could lead to immediate patient harm if acted upon.
   - `major`: Could significantly alter clinical understanding or decision-making.
   - `minor`: Factual inaccuracy with low potential for clinical impact.
5. For each hallucination, provide the exact text span from the Generated Medical Output and the conflicting or missing evidence from the Source Evidence.

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "hallucinations": [
    {
      "claim_text": "string",
      "hallucination_type": "fabricated_finding | incorrect_dosage | unsupported_diagnosis | missing_contraindication | demographic_error | temporal_error | omitted_material_fact",
      "severity": "critical | major | minor",
      "output_span": "string (exact text from generated output)",
      "source_evidence_span": "string (relevant excerpt from source evidence, or 'NO EVIDENCE FOUND')",
      "explanation": "string (brief clinical rationale for the flag)"
    }
  ],
  "overall_assessment": {
    "total_claims_extracted": number,
    "total_hallucinations_found": number,
    "critical_count": number,
    "major_count": number,
    "minor_count": number,
    "requires_human_review": boolean,
    "summary": "string (one-sentence clinical safety summary)"
  }
}

## CONSTRAINTS
- Only flag claims that are unsupported by the Source Evidence. Do not flag stylistic differences or paraphrasing that preserves clinical meaning.
- If the Source Evidence is insufficient to verify a claim, flag it as unsupported and note the evidence gap.
- Do not introduce external medical knowledge. Base all judgments solely on the provided Source Evidence.
- If no hallucinations are found, return an empty hallucinations array and set `requires_human_review` to false.
- Set `requires_human_review` to true if any hallucination has a severity of `critical` or `major`.

To adapt this template, replace [GENERATED_OUTPUT] with the medical text your system produced, [SOURCE_EVIDENCE] with the retrieved or provided clinical documents, and [CLINICAL_CONTEXT] with a brief description of the clinical scenario. The output schema is designed for direct ingestion by a validation harness. After receiving the JSON response, your application should automatically escalate any output where requires_human_review is true to a clinical reviewer and log all hallucinations for evaluation against a held-out set of annotated medical hallucination examples. Do not deploy this prompt without first measuring its recall and precision against a benchmark such as Med-HALT or a custom set of clinically annotated outputs.

This prompt operates as a post-generation guardrail, not a real-time interrupter. After a medical summary, note draft, or patient-facing output is generated, the hallucination detection prompt receives the full output alongside the source evidence (clinical notes, lab results, reference guidelines). The harness must enforce a strict contract: the prompt returns a structured JSON payload with flagged claims, severity labels, and evidence gaps. Do not rely on the model's self-assessment alone—the harness validates output schema compliance before any downstream action.

Pipeline integration steps: (1) Retrieve the source context that was used to generate the original output—this must be the exact evidence set, not a re-retrieved set that may differ. (2) Assemble the prompt with [GENERATED_OUTPUT], [SOURCE_EVIDENCE], [CLINICAL_DOMAIN], and [OUTPUT_SCHEMA] populated. (3) Call the model with response_format set to JSON Schema matching the expected hallucination report structure. (4) Validate the response: confirm all required fields exist, severity labels match the allowed enum, and each flagged span maps to a character offset or quote in the original output. (5) Route based on severity: critical or safety findings block the output and trigger human review; material findings queue for clinician review within a configurable SLA; minor findings log for offline audit without blocking the workflow. (6) Log the full prompt, response, validation result, and routing decision for audit trails.

Human review integration is mandatory for any clinical-facing system. The harness must surface flagged claims with their source evidence side by side in a review interface. Reviewers need one-click actions: confirm hallucination, override as acceptable paraphrase, or edit and release corrected output. Track reviewer decisions to measure the detector's precision and recall over time. Model choice matters: use a model with strong instruction-following and medical terminology awareness (GPT-4, Claude 3.5 Sonnet, or Med-PaLM variants). Avoid small or general-purpose models that conflate paraphrasing with hallucination. Eval calibration: run this prompt against a held-out set of outputs with clinician-annotated hallucination labels. Measure precision (flagged items that are true hallucinations) and recall (true hallucinations that were flagged). Tune the severity threshold in the routing logic based on your acceptable false-positive rate for blocking versus queuing. What breaks first: false positives on acceptable medical paraphrasing, missed hallucinations in rare disease contexts where the model lacks terminology, and schema non-compliance under high token pressure. Monitor all three in production dashboards.