Multi-Hop Query Plan Generation with Confidence Scores Prompt

Use this prompt when your RAG system must answer questions that require chained retrieval steps and you need to communicate uncertainty at each hop. The ideal user is an AI engineer or RAG architect building a production system where a single retrieval round cannot gather all necessary evidence. Typical scenarios include multi-hop fact verification, entity relationship traversal, and comparative analysis where intermediate answers are uncertain and downstream steps depend on them. The prompt requires a complex user question, access to retrieval tool definitions, and a defined confidence threshold below which the system should escalate or fall back.

Do not use this prompt for simple factual lookups, single-hop retrieval, or questions answerable from a single document. If your retrieval pipeline already has a fixed two-step pattern, a simpler query decomposition prompt will be faster and cheaper. This prompt adds latency and token cost proportional to the number of hops, so reserve it for questions where dependency chains are genuinely uncertain. The prompt assumes you have a retrieval function available as a tool; if you are generating queries for an external search system without tool-use feedback, use the Multi-Hop Query Chain Generation Prompt instead. For questions where confidence calibration is not required, the standard Multi-Hop Query Plan Generation Prompt without confidence scores will produce smaller, faster outputs.

Before deploying, calibrate the confidence scores against actual retrieval success rates in your corpus. A predicted confidence of 0.8 that maps to a 0.4 actual success rate will cause the system to proceed with bad intermediate evidence. Run the evaluation harness described in the implementation section against a golden dataset of known multi-hop questions with labeled dependency paths. If your latency budget is under 2 seconds end-to-end, consider whether a precomputed knowledge graph traversal can replace dynamic query planning. When the prompt is a fit, wire it into an agent loop that executes each hop, evaluates the retrieved evidence, and either continues, falls back, or escalates based on the predicted confidence and actual results.

This template is designed to be dropped into a RAG orchestration layer or an agent planning loop. It instructs the model to decompose a complex user question into a sequence of dependent retrieval steps, each annotated with a predicted confidence score. The critical addition is the fallback logic: for every hop where confidence is predicted to be below a configurable threshold, the model must propose an alternative query. This turns a static plan into a resilient execution graph, allowing your application to self-correct during retrieval without an immediate round-trip back to the planner.

codeCopy
SYSTEM:
You are a retrieval planning specialist. Your task is to decompose a complex user question into a multi-hop query plan. For each hop, predict a confidence score (0.0 to 1.0) that the retrieval will succeed. If the predicted confidence is below [CONFIDENCE_THRESHOLD], you must provide a fallback query. The plan must be a valid JSON object conforming to [OUTPUT_SCHEMA].

USER:
User Question: [USER_QUESTION]
Available Retrieval Tools: [AVAILABLE_TOOLS]
Context and Constraints: [CONTEXT]

Generate a multi-hop query plan following these rules:
1. Decompose the question into an ordered sequence of retrieval steps.
2. For each step, provide a `query` string, a `rationale`, and a `predicted_confidence` score.
3. If `predicted_confidence` < [CONFIDENCE_THRESHOLD], include a `fallback_query` array with one or more alternative queries.
4. Explicitly state the `dependency` for each hop (e.g., "none" or "hop_1.entity_id").
5. Include an `escalation_path` if all fallback queries for a critical hop are predicted to fail.

[OUTPUT_SCHEMA]
{
  "plan": [
    {
      "hop_id": "string",
      "query": "string",
      "rationale": "string",
      "dependency": "string | null",
      "predicted_confidence": "number",
      "fallback_queries": ["string"],
      "escalation_path": "string | null"
    }
  ]
}

To adapt this template, start by tuning [CONFIDENCE_THRESHOLD]. A value of 0.7 is a reasonable starting point for most knowledge bases, but you should calibrate this against historical retrieval precision in your system. The [AVAILABLE_TOOLS] placeholder should be populated with the names and descriptions of your actual retrieval functions or indexes (e.g., "vector_search", "keyword_search", "graph_traversal"). The [CONTEXT] field is where you inject user roles, time constraints, or domain-specific rules. After generating the plan, your harness must validate the JSON against [OUTPUT_SCHEMA] and check that every hop with a confidence below the threshold has a non-empty fallback_queries array. For high-stakes domains, route plans with any escalation_path populated directly to a human reviewer before execution begins.

This prompt is not a standalone chatbot interaction—it is a planning module inside a multi-hop RAG or agent pipeline. The prompt receives a complex user question and outputs a structured query plan with confidence scores, fallback queries, and escalation paths. The calling application must parse the JSON output, validate the plan structure, execute each hop in dependency order, and feed intermediate results back into subsequent retrieval steps. Treat the prompt as a state machine transition: it produces the next state (the plan), but the application owns execution, error handling, and evidence accumulation.

Validation and execution flow: Before executing any hop, validate the generated plan against a schema that checks for required fields (hop_id, query, depends_on, confidence, fallback_queries, escalation_path), non-empty query strings, valid dependency references (no self-loops or missing hop IDs), and confidence scores in the 0.0–1.0 range. Reject plans with circular dependencies or orphan hops. For each hop, execute the primary query against the target retrieval backend. If the retrieval returns zero results or the top result's relevance score falls below a configured threshold, automatically try the fallback queries in order. If all fallbacks fail, follow the escalation_path—this might mean routing to a human reviewer, switching to a broader retrieval index, or returning a partial answer with explicit uncertainty markers. Log every hop's retrieval latency, result count, and whether a fallback was triggered; these logs become the ground truth for calibrating the confidence scores the prompt predicts.

Confidence calibration loop: The prompt predicts a confidence score for each hop, but that prediction is only useful if it correlates with actual retrieval success. Implement a periodic calibration job that compares predicted confidence scores against observed retrieval outcomes (success/failure, relevance scores, fallback trigger rates) across a sample of production traces. Use this data to adjust the threshold at which you trust a hop's predicted confidence versus requiring additional verification. If predicted confidence consistently overestimates success, add a calibration offset or surface the miscalibration in the prompt's own output by including a calibration_note field. Model choice: This prompt benefits from models with strong planning and structured output capabilities. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro are good starting points. Avoid smaller models (under 7B parameters) for production use unless you've validated their plan coherence on your specific query distribution—they tend to produce plans with missing dependencies or hallucinated hop IDs.

Retries and failure modes: The most common production failure is a plan that looks structurally valid but contains a bridge entity hallucination—a hop that depends on retrieving an entity that doesn't exist in your corpus. Mitigate this by checking whether intermediate entities extracted from hop N-1 actually appear in your knowledge base before using them to construct hop N's query. If an entity is absent, halt the chain and either request a revised plan from the prompt (passing the failure context back as [CONTEXT]) or escalate. Set a maximum hop depth (we recommend 5 hops) and a total latency budget (e.g., 3 seconds) to prevent runaway chains. If the plan exceeds either limit, fall back to a simpler single-hop retrieval strategy and flag the interaction for review. Wire the prompt into your observability stack: log the full plan, per-hop outcomes, fallback usage, and final answer quality so you can trace failures back to planning errors versus retrieval gaps.