Evidence Gap Report Generation Prompt Template

This prompt is designed for AI engineers and RAG pipeline operators who need a structured, auditable artifact that documents the boundaries of retrieved knowledge. The core job-to-be-done is not to answer a user's question, but to produce a machine-readable report detailing what the context covers, what information is missing, the impact of those gaps on answer quality, and recommended remediation steps. The ideal user is building an internal observability system, a retrieval debugging workflow, or a human review queue where traceability and structured evidence are required. The prompt expects a user question and a set of retrieved context chunks as input, and it outputs a gap report with explicit categories, severity assessments, and actionable next steps.

This prompt should be deployed in backend diagnostic pipelines, not in customer-facing latency paths. Use it when you need to log evidence gaps for dashboarding, trigger re-retrieval with modified queries, or populate a review queue for knowledge base curators. The output schema is designed to be consumed by downstream systems—logging platforms, alerting rules, and retrieval optimization scripts—rather than by end users. Do not use this prompt for real-time answer generation, for directly responding to users, or as a simple pass/fail sufficiency check. It is intentionally verbose and structured for auditability, which makes it unsuitable for low-latency conversational flows. Pair it with the Pre-Answer Sufficiency Check Prompt Template if you need a lightweight gate before generation, and reserve this report for post-hoc analysis, pipeline improvement, and compliance logging.

Before wiring this prompt into your application, ensure you have a clear contract for the input context format—typically an array of objects with chunk_id, text, and optional metadata fields. The prompt's value scales with the quality of your retrieval pipeline: if your retriever consistently returns irrelevant chunks, the gap report will correctly identify widespread insufficiency but won't fix the underlying retrieval problem. Start by running this prompt against a golden dataset of known-sufficient and known-insufficient context sets to calibrate your expectations. Then integrate the structured output into your observability stack, setting alerts on recurring gap categories or high-severity missing entities. The next step after reading this section is to copy the prompt template, adapt the output schema to match your logging infrastructure, and build a validation harness that checks for schema compliance before the report enters your review queue.

This template is the core instruction set for an LLM to produce an auditable Evidence Gap Report. It is designed to be pasted directly into a system or user message. The prompt forces the model to separate what the context covers from what is missing, assess the impact of each gap, and recommend concrete remediation steps. All dynamic inputs are represented as square-bracket placeholders. Replace these with live data from your retrieval pipeline, application state, and output schema requirements before sending the request.

textCopy
You are an evidence auditor for a Retrieval-Augmented Generation (RAG) system. Your task is to analyze the provided context against the user's question and produce a structured Evidence Gap Report. You must strictly separate facts present in the context from information that is missing, unclear, or contradictory.

## INPUTS
**User Question:** [USER_QUESTION]
**Retrieved Context:** [RETRIEVED_CONTEXT]
**Required Output Schema:** [OUTPUT_SCHEMA]
**Confidence Threshold:** [CONFIDENCE_THRESHOLD]

## INSTRUCTIONS
1.  **Parse the Question:** Decompose the user's question into discrete information needs or sub-questions.
2.  **Map Evidence:** For each information need, identify which parts of the retrieved context provide supporting, contradictory, or no evidence.
3.  **Identify Gaps:** For each unmet information need, classify the gap type:
    - `missing_fact`: The context does not contain the required information.
    - `stale_information`: The context contains information that may be outdated relative to the question's temporal frame.
    - `source_conflict`: Multiple sources in the context provide contradictory information.
    - `insufficient_detail`: The context touches on the topic but lacks the specificity required to answer.
    - `entity_not_found`: A specific entity mentioned in the question is absent from the context.
4.  **Assess Impact:** For each gap, rate the impact on answer quality as `critical`, `high`, `medium`, or `low`.
5.  **Recommend Remediation:** For each gap, suggest a concrete action to resolve it (e.g., a specific query rewrite, a new data source to query, a metadata filter to apply).
6.  **Generate Report:** Output a single JSON object conforming to the provided [OUTPUT_SCHEMA]. Do not include any text outside the JSON object.

## CONSTRAINTS
- Do not invent information to fill gaps. If the context is silent, the gap status is `missing_fact`.
- If the overall confidence in the available evidence is below the [CONFIDENCE_THRESHOLD], set the top-level `answerable` field to `false`.
- Cite specific context snippets by their `source_id` when mapping evidence.

To adapt this template, start by defining your [OUTPUT_SCHEMA]. A robust schema should include a top-level answerable boolean, an array of information_needs with their status and evidence_map, and a separate gaps array containing the structured gap objects. The [CONFIDENCE_THRESHOLD] is a critical business logic parameter that should be tuned in your eval harness, not hardcoded in the prompt. For high-stakes domains, always route reports where answerable is false or where critical gaps exist to a human review queue before any downstream action is taken.

The Evidence Gap Report prompt is designed to be called after retrieval but before answer generation in a RAG pipeline. It acts as a diagnostic gate that produces a structured JSON report. The primary integration point is a post-retrieval hook that receives the user's question and the top-k retrieved chunks, then decides whether to proceed with generation, trigger re-retrieval, or escalate for human review. This prompt is not a user-facing answer generator—it is an internal observability and control-plane component.

Wire the prompt into your application with a strict JSON output contract. The model must return a valid JSON object matching the [OUTPUT_SCHEMA] you define, typically including fields like coverage_summary, identified_gaps (array of objects with gap_description, severity, impact_on_answer), overall_sufficiency_score, and recommended_action (enum: proceed, re_retrieve, escalate). Implement a validation layer that parses the JSON response, checks for required fields, validates enum values, and confirms the overall_sufficiency_score is within the expected numeric range. If validation fails, retry once with the validation error message appended to the prompt as feedback. After two consecutive failures, log the raw output and escalate to a human review queue rather than silently proceeding with a potentially misleading gap report.

For model choice, prefer models with strong instruction-following and structured output capabilities (e.g., GPT-4o, Claude 3.5 Sonnet) because the gap report requires precise enumeration of missing evidence rather than fluent prose. Set temperature low (0.0–0.2) to maximize consistency across repeated runs on the same input. If your pipeline uses a cheaper model for answer generation, you can still route the gap report to a more capable model since this step is a gating decision that affects downstream safety and correctness. Log every gap report alongside the retrieval query, retrieved chunk IDs, and the final answer (if generated) to create an audit trail. This enables retrospective analysis of whether gaps were correctly identified and whether proceed decisions led to hallucinated answers.

The recommended_action field is your primary routing signal. Map proceed to your answer generation step, re_retrieve to a retrieval expansion module (which may rewrite the query, broaden search parameters, or query alternative indexes), and escalate to a human review queue with the full gap report attached. For re_retrieve actions, implement a loop limit (maximum 2–3 retrieval rounds) to prevent infinite cycles when the knowledge base genuinely lacks the needed information. After the loop limit is exhausted, force an escalate path. Test the harness with a golden dataset of questions where you know the knowledge base's coverage boundaries—questions fully covered, partially covered, and entirely outside scope—and measure whether the gap report's recommended_action aligns with ground truth. Track false-positive proceed decisions (gaps missed) and false-positive escalate decisions (unnecessary human review) as your primary quality metrics.