Clinical Safety Confidence Prompt for Healthcare RAG

This prompt is designed for clinical informatics teams deploying RAG systems in healthcare settings where a wrong answer can cause patient harm. It is not a general-purpose medical Q&A prompt. Its specific job is to act as a safety gate after retrieval and before answer generation, producing a structured safety assessment with explicit abstention triggers for diagnosis, drug interaction, and treatment questions. The ideal user is an AI engineer or clinical informaticist who has already built a retrieval pipeline over a trusted knowledge base (e.g., FDA labels, clinical guidelines, institutional protocols) and needs a machine-readable escalation signal before any answer reaches a clinician or patient.

Use this prompt when your system must reliably distinguish between safe informational queries ("What is the mechanism of action of metformin?") and dangerous clinical-decision queries ("Should this patient with CKD be on metformin?"). It assumes you have retrieved clinical context and need a safety gate, not a content generator. The prompt outputs a structured payload with a safety_decision (PROCEED, CAUTION, ESCALATE, BLOCK), a risk_category, and explicit abstention_triggers. This output is designed to be consumed by application logic that routes PROCEED responses to an answer-generation model, ESCALATE responses to a human review queue, and BLOCK responses to a refusal message. Do not use this prompt for patient-facing chatbots without a human-in-the-loop review step for any output that is not BLOCK. Do not use it as a substitute for clinical judgment or as a diagnostic tool.

Before integrating this prompt, ensure your retrieval pipeline is grounded in a controlled, trusted knowledge base. The safety assessment is only as reliable as the context it evaluates. Pair this prompt with an eval harness that measures over-refusal and under-refusal rates against a golden dataset of borderline clinical queries. Monitor the escalation rate in production; a sudden spike may indicate retrieval degradation or a change in the underlying model's risk sensitivity. The next section provides the copy-ready prompt template you can adapt to your specific clinical domain and risk tolerance.

This prompt template is designed to be placed in your system prompt or as a user message template within a healthcare RAG pipeline. It instructs the model to act as a clinical safety gate, evaluating a proposed answer against retrieved evidence and producing a structured safety assessment. The output is not the final answer to the user; it is a machine-readable decision payload that your application must parse to determine whether to display the answer, escalate for human review, or refuse with a safe message. The template uses square-bracket placeholders that your application must populate before each call: the user's original query, the proposed answer generated by your primary model, the retrieved clinical context, and your organization's specific safety policy rules.

textCopy
You are a clinical safety reviewer for a healthcare question-answering system. Your role is to assess whether a proposed answer is safe to show to a clinician or patient based on the retrieved clinical evidence and the organization's safety policy.

## INPUTS
- USER_QUERY: [USER_QUERY]
- PROPOSED_ANSWER: [PROPOSED_ANSWER]
- RETRIEVED_CONTEXT: [RETRIEVED_CONTEXT]
- SAFETY_POLICY: [SAFETY_POLICY]

## SAFETY POLICY RULES
[SAFETY_POLICY]

## OUTPUT SCHEMA
Return a single JSON object with exactly these fields:
{
  "safety_decision": "SAFE" | "REFUSE" | "ESCALATE",
  "confidence_score": 0.0-1.0,
  "abstention_triggers": ["diagnosis" | "drug_interaction" | "treatment_recommendation" | "dosage" | "prognosis" | "none"],
  "risk_flags": ["unsupported_claim" | "missing_evidence" | "contradictory_sources" | "outdated_evidence" | "patient_specific_advice" | "none"],
  "escalation_reason": "string or null if not escalated",
  "evidence_sufficiency": "SUFFICIENT" | "PARTIAL" | "INSUFFICIENT",
  "caveats": ["string caveats to attach if answer is shown"],
  "human_review_priority": "ROUTINE" | "URGENT" | "CRITICAL" | null
}

## DECISION RULES
1. Set safety_decision to "REFUSE" if the query or proposed answer involves diagnosis, drug interactions, treatment recommendations, dosage calculations, or prognosis for a specific patient, unless the RETRIEVED_CONTEXT contains explicit, current, authoritative guidance that directly supports the answer AND the SAFETY_POLICY permits it.
2. Set safety_decision to "ESCALATE" if evidence is partial, sources conflict, or the query requires clinical judgment beyond what the context supports. Include the specific reason in escalation_reason.
3. Set safety_decision to "SAFE" only when the proposed answer is strictly grounded in the retrieved context, makes no clinical claims beyond the evidence, and complies with all SAFETY_POLICY rules.
4. Populate abstention_triggers with every category that triggered a REFUSE or ESCALATE decision. Use ["none"] if the answer is SAFE.
5. Populate risk_flags with every risk factor detected, even if the answer is SAFE. Use ["none"] if no risks are present.
6. Set human_review_priority based on potential patient harm: "CRITICAL" for life-threatening scenarios, "URGENT" for time-sensitive clinical decisions, "ROUTINE" for non-urgent review, null if SAFE.
7. Confidence_score must reflect your certainty in the safety assessment itself, not the certainty of the proposed answer. A score below 0.7 should default the decision to ESCALATE.

## CONSTRAINTS
- Do not generate any text outside the JSON object.
- Do not modify, rewrite, or improve the proposed answer.
- Do not provide medical advice or clinical interpretation beyond the safety assessment.
- If the RETRIEVED_CONTEXT is empty or irrelevant, set evidence_sufficiency to "INSUFFICIENT" and safety_decision to "REFUSE".
- Always err on the side of caution. When in doubt, ESCALATE.

To adapt this template for your system, start by defining your SAFETY_POLICY as a separate, version-controlled configuration block that enumerates the specific clinical domains, query types, and evidence standards that trigger refusal or escalation. Replace the placeholder with a concise, bulleted policy statement that your clinical governance team has approved. Wire the output JSON into a post-processing gate: parse safety_decision, and route SAFE answers to the user, REFUSE decisions to a canned safe-response message, and ESCALATE decisions to your clinical review queue with the full payload attached. Log every safety assessment alongside the query, answer, and context for auditability. Run this prompt against a golden dataset of known-safe and known-unsafe query-answer pairs before deployment, measuring false-acceptance and false-refusal rates against your organization's risk tolerance. Never bypass this gate for queries that touch the abstention trigger categories, even if your primary model expresses high confidence.

This prompt is not a standalone chatbot. It is a safety-critical component inside a larger clinical informatics pipeline. The implementation harness must treat the model's output as a structured signal, not a final answer. Before any text reaches a clinician or patient, the application layer must validate the JSON schema, check the abstention flags, and route the result through the appropriate escalation path. The prompt's primary job is to produce a machine-readable safety assessment; the harness's job is to enforce what happens next.

Wire the prompt into a RAG pipeline that retrieves evidence from an approved clinical knowledge base (e.g., drug interaction databases, treatment guidelines, curated medical references). The prompt expects a [CONTEXT] block containing retrieved passages with source metadata and a [QUERY] block with the user's question. The application should construct these inputs programmatically, never from raw user text alone. After generation, validate the output against a strict JSON schema that requires the safety_tier field (Safe / Caution / Abstain), a confidence_score between 0 and 1, a reason_code from the allowed enum, and a clinical_review_required boolean. Reject and retry any response that fails schema validation. Log every output—including the prompt version, model, retrieved context IDs, and validation result—to an immutable audit store.

For high-risk tiers, the harness must enforce hard gates. If safety_tier equals Abstain or clinical_review_required is true, the system must not display the answer to the end user. Instead, route the query, retrieved evidence, and model output to a clinical review queue with a priority derived from the reason_code (e.g., diagnosis_request or drug_interaction_question triggers immediate escalation). For Caution tier outputs, the application should surface the answer only after prepending a mandatory disclaimer and linking to the source evidence. Use a separate evaluation harness to run this prompt against a golden dataset of known safe, borderline, and dangerous queries, measuring abstention accuracy, over-refusal rate, and schema compliance before any production deployment. Never rely on the model's text alone to gate safety—the application code must be the final enforcement layer.