Retry with Expanded Retrieval Window Prompt

This prompt is designed for a specific failure mode in production RAG systems: the initial retrieval step returns evidence that is too thin, off-topic, or insufficient to generate a well-grounded answer. Instead of allowing the model to hallucinate or returning a generic failure message, this prompt orchestrates a structured recovery step. It instructs the system to re-execute retrieval with a deliberately broadened scope—such as wider date ranges, relaxed metadata filters, a higher top-k value, or expanded query variants—and then regenerates the answer from this richer evidence set. The primary job-to-be-done is to increase the recall of potentially relevant documents before the final generation step, giving the model a better chance to synthesize a faithful, cited response.

The ideal user for this playbook is an AI engineer or platform developer who has direct control over a tunable retrieval pipeline. You must be able to programmatically adjust parameters like top_k, date filters, similarity thresholds, or query strings. This prompt is not a replacement for a dedicated query-rewriting step; it assumes you have already attempted a targeted retrieval and that the failure is due to overly restrictive parameters, not a poorly formed initial query. Use this prompt when your system's architecture allows for a low-cost retry loop before escalating to a human or returning a final abstention. It is a tactical recovery tool, not a primary retrieval strategy.

There are clear situations where this prompt should not be used. Do not deploy it in systems where the retrieval scope is fixed, non-configurable, or where a second retrieval pass would be prohibitively expensive or slow for the user experience. It is also the wrong tool if the underlying problem is a fundamental lack of relevant documents in your corpus; expanding a search over an empty or irrelevant knowledge base will only waste resources and delay a necessary abstention. Furthermore, avoid using this prompt as a bandage for a broken initial retrieval pipeline. If your first-pass retrieval consistently fails, invest in improving query rewriting, embedding quality, or chunking strategy rather than relying on this recovery loop as a permanent fix.

This prompt template is designed to be injected into your RAG pipeline's retry handler after the initial retrieval pass returns evidence that fails a sufficiency check. It instructs the model to analyze why the original evidence was insufficient, formulate broader retrieval queries, and regenerate a grounded answer from the expanded result set. The square-bracket placeholders are meant to be populated by your application layer before the prompt reaches the model—never pass unresolved tokens.

textCopy
You are a retrieval-augmented generation system recovering from insufficient evidence.

[ORIGINAL_QUERY]
[ORIGINAL_ANSWER]

[INITIAL_RETRIEVED_CHUNKS]

[SUFFICIENCY_FAILURE_REASON]

Your task:
1. Analyze the original answer and identify every factual claim that lacks adequate support from [INITIAL_RETRIEVED_CHUNKS].
2. Generate [EXPANDED_QUERY_COUNT] new retrieval queries that broaden the search scope. Consider:
   - Synonym expansion for key terms
   - Removing restrictive filters that may have excluded relevant documents
   - Searching for adjacent concepts or supporting evidence
   - Querying for counter-evidence or alternative perspectives
3. After expanded retrieval completes, you will receive [EXPANDED_RETRIEVED_CHUNKS].
4. Regenerate the answer using the expanded evidence. Apply these constraints:
   - Every factual claim must be directly traceable to at least one chunk in [EXPANDED_RETRIEVED_CHUNKS].
   - Use inline citations in the format [SOURCE_ID] for each claim.
   - If a claim from the original answer still lacks support after expanded retrieval, remove it or replace it with an explicit abstention statement.
   - Do not introduce claims that cannot be sourced to the expanded evidence.
5. Output a JSON object with the following schema:
   {
     "expanded_queries": ["string"],
     "insufficient_claims_removed": ["string"],
     "regenerated_answer": "string with [SOURCE_ID] citations",
     "abstention_notices": ["string"],
     "coverage_improvement": "string explaining what the expanded retrieval added"
   }

[OUTPUT_VALIDATION_RULES]
[RETRIEVAL_PARAMETER_OVERRIDES]

To adapt this template, start by defining your sufficiency threshold in the application layer—common signals include citation coverage below 80%, confidence scores under 0.7, or explicit abstention triggers. Populate [SUFFICIENCY_FAILURE_REASON] with a structured description of what went wrong, such as "3 of 7 claims lack source support" or "retrieved chunks cover only product A, but answer references product B." The [RETRIEVAL_PARAMETER_OVERRIDES] field should contain the actual parameter changes your retriever will apply, such as top_k: 20, similarity_threshold: 0.5, or hybrid_search: true. Wire the output through a JSON schema validator before accepting the regenerated answer, and log the coverage_improvement field to track whether expanded retrieval actually resolved the gap. For high-stakes domains, route outputs with remaining abstention notices to human review rather than surfacing them directly to users.

This prompt is not a standalone fix; it is a recovery step inside a broader RAG harness. The harness must detect when the initial answer is insufficient—typically via a citation completeness check, a confidence score below a defined threshold, or a validator that flags unsupported claims—and only then trigger the expanded retrieval retry. The harness should never call this prompt blindly on every request, because expanding the retrieval window increases latency, token consumption, and the risk of introducing lower-quality or contradictory sources. Instead, gate the retry behind a clear sufficiency check: if the initial answer already meets your citation coverage, confidence, and grounding requirements, skip the retry entirely.

When the retry is triggered, the harness must programmatically adjust the retrieval parameters before re-invoking the model. This means widening the top_k value, relaxing metadata filters, lowering the similarity threshold, or switching from dense to hybrid retrieval—whatever your retriever supports. The harness should capture the original retrieval parameters and the expanded parameters in the trace log so operators can later analyze whether the expansion actually improved evidence quality. After re-retrieval, the harness must pass both the original user query and the expanded context into this prompt, along with any failure reason from the initial validation step (e.g., '3 of 7 claims lacked citations' or 'confidence score 0.62 below threshold 0.85'). The model needs to know why it is being asked to regenerate, not just that it is.

After the model returns a regenerated answer, the harness must re-run the same validation suite that flagged the original failure: citation coverage checks, source-to-claim alignment verification, quote fidelity scoring, or whatever eval gates your system uses. If the regenerated answer still fails, the harness should not retry indefinitely. Implement a retry budget—typically 1-2 expanded-retrieval attempts—and if the answer still does not meet the quality bar, escalate to a fallback path: return the best available answer with explicit uncertainty flags, trigger a human review queue, or respond with a structured abstention that explains what evidence was found and why it was insufficient. Log every attempt, the retrieval parameters used, the validation scores, and the final disposition so that the retry loop is fully observable and debuggable in production.

Model choice matters here. The expanded retrieval window may produce significantly more context, so use a model with a context window large enough to hold the broader evidence set plus the prompt instructions and output schema. If the expanded context still overflows the window, the harness must apply a context-packing strategy—prioritize the highest-relevance passages, summarize lower-ranked sources, or chunk the evidence and run multiple passes—before calling this prompt. Finally, ensure that the harness records the retrieval expansion as a distinct trace event with its own span ID, so that monitoring dashboards can distinguish between first-attempt answers and retry-with-expanded-retrieval answers. This separation is critical for measuring the true cost and effectiveness of the recovery strategy over time.