RAG Hallucination Risk Assessment Prompt

This prompt is designed for AI safety engineers and output validation pipeline builders who need a structured, automated gate before a user-facing answer is generated. Its primary job is to assess the risk of hallucination given a specific user query and a set of retrieved evidence chunks. It does not generate an answer; instead, it produces a risk level and a breakdown of contributing factors such as gap severity, source ambiguity, and query complexity. The ideal user is integrating this into a production RAG pipeline where the cost of a confident but wrong answer is high, such as in customer support, clinical documentation, or financial analysis.

Deploy this prompt as a synchronous check between the retrieval step and the answer generation step. The required inputs are a [USER_QUERY] and a [RETRIEVED_CONTEXT] array. The output is a structured JSON object containing a [RISK_LEVEL] (e.g., LOW, MEDIUM, HIGH, CRITICAL) and a [RISK_FACTORS] list. Use this output to route the request: LOW risk can proceed directly to answer generation, MEDIUM might trigger a re-retrieval or a partial answer with caveats, and HIGH or CRITICAL should escalate for human review or return a refusal message. This prompt is not a replacement for a full grounding verification check on the final generated answer; it is a pre-emptive safety filter.

Do not use this prompt when the user query is trivial, the operational cost of refusal is higher than the cost of a minor hallucination, or when the retrieval system is known to be highly unreliable. In these cases, the risk assessment itself becomes a bottleneck or a source of noise. Avoid using this prompt as a standalone quality score without downstream routing logic; its value is realized only when the risk level actively changes the system's behavior. The next step after implementing this gate is to pair it with a 'Missing Context Request Generation Prompt' for MEDIUM risk scenarios, ensuring the system can actively work to de-risk the query before giving up.

This prompt template is designed to be used as a pre-generation safety gate. It takes a user query and the retrieved evidence set and produces a structured risk assessment before any answer is shown to the user. The output includes a risk level, contributing factors, and a recommendation for whether to proceed, escalate, or refuse. Use this template when the cost of a hallucinated answer is high—customer-facing support, clinical summaries, financial analysis, or any workflow where user trust depends on accuracy.

textCopy
You are a hallucination risk assessor for a Retrieval-Augmented Generation (RAG) system. Your job is to evaluate whether the available evidence is sufficient to answer the user's query without hallucination. Do not answer the query. Only assess the risk.

## INPUT

**User Query:**
[USER_QUERY]

**Retrieved Evidence:**
[RETRIEVED_EVIDENCE]

**Risk Tolerance:** [RISK_TOLERANCE]

## ASSESSMENT DIMENSIONS

Evaluate the following dimensions and assign a severity level (NONE, LOW, MEDIUM, HIGH, CRITICAL) for each:

1. **Evidence Gaps:** Are there specific facts, entities, dates, or claims in the query that have no supporting evidence?
2. **Source Ambiguity:** Do multiple sources conflict, or is the evidence vague, outdated, or from low-authority sources?
3. **Query Complexity:** Does the query require multi-hop reasoning, temporal comparisons, or synthesis across many passages?
4. **Inference Distance:** How many inferential steps separate the raw evidence from a complete answer?
5. **Coverage Completeness:** What percentage of the query's information needs are directly addressed by the evidence?

## OUTPUT SCHEMA

Return a JSON object with this exact structure:

{
  "risk_level": "LOW" | "MEDIUM" | "HIGH" | "CRITICAL",
  "risk_score": 0.0-1.0,
  "dimension_assessments": [
    {
      "dimension": "string",
      "severity": "NONE" | "LOW" | "MEDIUM" | "HIGH" | "CRITICAL",
      "explanation": "string"
    }
  ],
  "contributing_factors": ["string"],
  "recommendation": "PROCEED" | "PROCEED_WITH_CAVEATS" | "ESCALATE" | "REFUSE",
  "recommendation_rationale": "string",
  "missing_information_summary": "string or null"
}

## CONSTRAINTS

- Do not answer the user's query.
- Do not fabricate evidence or fill gaps with your own knowledge.
- If evidence is missing, state what is missing explicitly.
- If sources conflict, describe the conflict rather than choosing sides.
- For HIGH or CRITICAL risk, the recommendation must be ESCALATE or REFUSE.
- Base every assessment strictly on the provided evidence.

Adaptation guidance: Replace [USER_QUERY] with the raw user input. [RETRIEVED_EVIDENCE] should contain the full text of retrieved passages, ideally with source identifiers and retrieval scores. [RISK_TOLERANCE] accepts values like LOW, MEDIUM, or HIGH and should be set based on the application's tolerance for hallucination—clinical and financial systems should use LOW. For production use, add [EXAMPLES] with 2-3 annotated query-evidence pairs showing correct risk assessments to improve consistency. If your system uses a specific output schema language, replace the JSON block with your schema definition. For high-throughput systems, consider adding a [CONSTRAINTS] block that limits the response to 200 tokens and removes the explanation fields to reduce latency.

This prompt is designed to operate as a pre-generation safety gate in a RAG pipeline. It should run after retrieval and evidence assembly but before the final answer generation step. The model receives the user query, the retrieved context set, and any metadata about source recency or authority. It returns a structured risk assessment that your application can use to decide whether to proceed with answer generation, request more evidence, escalate for human review, or return a refusal to the user. The prompt is not a post-hoc hallucination detector—it predicts risk before the answer exists, which makes it valuable for preventing confident wrong answers rather than catching them afterward.

Wire the prompt into your application as a synchronous pre-check with a strict timeout and a deterministic fallback. On the happy path, parse the JSON output and read the risk_level field. Map risk levels to routing decisions: low proceeds directly to answer generation; medium may proceed with additional grounding instructions injected into the answer-generation prompt; high triggers a re-retrieval attempt or clarification request; critical escalates to a human review queue or returns a refusal message. Always log the full risk assessment payload alongside the query, retrieval set, and eventual answer for audit and evaluation. If the model returns malformed JSON, apply a repair prompt or fall back to a conservative high risk classification rather than silently proceeding.

Validation and retry logic is essential because this prompt sits on the critical path. Validate that the output contains a recognized risk_level value from the allowed set (low, medium, high, critical), that contributing_factors is a non-empty array, and that each factor includes both a factor name and a severity rating. If validation fails, retry once with the same input and a stronger format reminder appended to the prompt. If the retry also fails, log the failure, classify the risk as high, and route to human review. For model choice, prefer a fast, instruction-following model with strong JSON output discipline—GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are suitable. Avoid using a small or local model for this gate unless you have calibrated its risk assessment accuracy against a labeled dataset, because false-negative low risk classifications will let hallucinations through to users.

Evaluation and calibration should happen before production deployment. Build a test set of 50–100 query-and-context pairs with known hallucination outcomes from your own system. Run the risk assessment prompt against this set and measure whether high and critical risk classifications correlate with actual hallucination events. Tune the routing thresholds based on your tolerance for false positives (unnecessary refusals) versus false negatives (hallucinations reaching users). In high-stakes domains such as healthcare, legal, or finance, bias toward over-classifying risk and routing to human review. In lower-stakes applications, you may allow medium risk to proceed with stronger grounding instructions. Document your threshold decisions and re-evaluate them whenever you change retrieval logic, knowledge base content, or the underlying model.