Multi-Hop Reasoning Prompt over Retrieved Documents

This prompt is for complex questions that cannot be answered by a single retrieved passage. Its job is to force the model to chain evidence across multiple documents, producing a step-by-step reasoning trace with intermediate findings, source references, and a final synthesized answer. The ideal user is an AI engineer or RAG system builder whose pipeline returns several related but non-overlapping passages, and the user question requires connecting facts across them—for example, 'What was the revenue impact of the acquisition mentioned in the Q3 report and the subsequent restructuring outlined in the internal memo?' This is not a prompt for simple factoid lookup, single-document summarization, or conversational chit-chat. It belongs in workflows where auditability of the reasoning path matters as much as the final answer, such as legal research, financial analysis, or clinical evidence review.

Before using this prompt, you must have a retrieval step that surfaces a set of candidate documents. The prompt expects a [CONTEXT] variable containing multiple passages with unique identifiers. It also requires a [QUESTION] that explicitly demands connecting information across sources. The output contract is strict: the model must produce a structured reasoning chain, not just a fluent answer. You should wire in validation checks that parse the output for intermediate steps, verify that each claim cites a source ID present in the input, and flag any 'dead ends' where the model states it cannot connect two pieces of evidence. If your use case involves regulated or high-risk domains, add a human review step before the final answer is shown to users. Do not use this prompt when a single document contains the full answer, when latency constraints prohibit multi-step reasoning, or when the retrieval set is too noisy to support logical chaining.

Start by testing this prompt against a golden dataset of 10-20 multi-hop questions where you know the correct evidence chain. Measure two things: answer correctness and reasoning trace validity. If the model skips intermediate steps or hallucinates connections between unrelated passages, tighten the [CONSTRAINTS] section to require explicit 'CONNECT' statements between each hop. Avoid the temptation to use this prompt as a general-purpose RAG template; its value is in forcing explicit, auditable reasoning for questions that genuinely require it. For simpler lookups, use a standard RAG citation prompt instead.

This template orchestrates a multi-hop reasoning workflow over a set of retrieved documents. It instructs the model to decompose a complex question into sub-questions, answer each using specific passages, and chain intermediate findings into a final answer. The prompt enforces explicit source grounding for every reasoning step, making it suitable for audit-heavy domains where traceability is non-negotiable. Use this template when a single document cannot answer the question and the model must synthesize information across two or more passages.

textCopy
You are a precise reasoning assistant. Your task is to answer a complex question by chaining evidence across multiple retrieved documents. Follow these rules exactly.

## INPUT
Question: [USER_QUESTION]
Retrieved Documents:
[DOCUMENTS]

## CONSTRAINTS
- Decompose the question into sub-questions that must be answered in sequence. Each sub-question should depend on the answer to the previous one.
- For each sub-question, identify the specific document(s) and passage(s) that provide the answer. Quote the relevant text.
- If a sub-question cannot be answered from the provided documents, stop and report a dead-end. Do not guess.
- After answering all sub-questions, synthesize a final answer that explicitly references the evidence chain.
- If the final answer requires information not present in the documents, state that the answer is incomplete and explain what is missing.

## OUTPUT FORMAT
Return a JSON object with this exact structure:
{
  "reasoning_trace": [
    {
      "step": 1,
      "sub_question": "string",
      "answer": "string or null",
      "source_document_ids": ["string"],
      "quoted_evidence": ["string"],
      "confidence": "high|medium|low",
      "dead_end": false
    }
  ],
  "final_answer": "string",
  "answerable": true,
  "missing_information": ["string"]
}

## EXAMPLES
[FEW_SHOT_EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

Adaptation notes: Replace [USER_QUESTION] with the complex question requiring multi-hop reasoning. [DOCUMENTS] should contain the full set of retrieved passages, each with a unique identifier and the passage text. [FEW_SHOT_EXAMPLES] is optional but strongly recommended for production—include 1-3 examples showing a complete reasoning trace with dead-end handling. Set [RISK_LEVEL] to high if the domain is regulated (finance, healthcare, legal) to trigger additional validation and human review downstream. If your application cannot parse structured JSON reliably, replace the output format with a markdown template, but expect more post-generation repair work. Always validate that every step in the reasoning trace cites a document that actually exists in the input set—this is the most common failure point in production.

The multi-hop reasoning prompt is not a standalone artifact—it is a component inside a retrieval-augmented generation pipeline that must be orchestrated with care. In production, this prompt typically sits after a retrieval step that returns a candidate document set and before a final answer synthesis or user-facing response. The harness is responsible for assembling the prompt with the correct [QUESTION], [RETRIEVED_DOCUMENTS], and [REASONING_INSTRUCTIONS], then parsing the model's structured reasoning trace for downstream use. Because multi-hop reasoning can produce dead ends, circular logic, or unsupported leaps, the harness must include validation checks that inspect the reasoning chain before it reaches the user or feeds into the next pipeline stage.

Wire the prompt into your application as a dedicated reasoning service with clear input and output contracts. On the input side, accept a user question and a list of retrieved document objects, each containing at minimum a doc_id, content, and optional source_metadata. Before calling the model, validate that the retrieved set is non-empty and that total token length does not exceed your context budget minus the prompt template overhead. On the output side, parse the model response against a strict schema that includes reasoning_steps (an ordered list of hop objects with step_number, finding, source_doc_ids, and confidence), intermediate_conclusions, dead_ends_detected (with explanation), and final_answer. If the model fails to produce valid JSON, retry once with a repair prompt that includes the raw output and the expected schema. If the second attempt fails, log the failure and escalate to a fallback answer path—do not loop indefinitely.

Model choice matters for multi-hop reasoning workloads. Claude 3.5 Sonnet and GPT-4o are strong defaults for complex reasoning traces, but test your specific document complexity and hop depth. For cost-sensitive deployments, consider routing simpler questions (fewer than two hops) to a faster model and reserving the full multi-hop prompt for questions flagged by a lightweight classifier. Instrument every call with trace metadata: prompt_version, model_id, retrieval_set_id, hop_count, dead_end_count, and latency_ms. This data is essential for debugging reasoning failures and tuning retrieval quality. If your use case involves regulated domains, add a human review gate that surfaces reasoning traces with confidence below a configurable threshold or dead_ends_detected that were not resolved.

The most common production failure mode is the model inventing connections between documents that do not actually support the claimed relationship. Mitigate this by implementing a post-reasoning verification step: for each hop's source_doc_ids, check that the cited documents exist in the retrieved set and that the claimed finding is textually supported. A lightweight NLI model or a second LLM call with a focused verification prompt can catch fabricated hops before they propagate. If verification fails for any hop, flag the entire reasoning trace for review and do not surface the final_answer to the user without a caveat. Next, build an eval harness that tests the prompt against a golden dataset of multi-hop questions with known answer paths, measuring hop accuracy, dead-end detection recall, and final answer correctness. Run these evals on every prompt change before deployment.