Multi-Hop Citation Chain Validation Prompt

This prompt is designed for RAG developers and quality engineers who need to validate complex, multi-step reasoning chains in production traces. It is used when a generated answer synthesizes information from multiple retrieved sources, and you need to verify that each inferential step is supported by evidence. The core job-to-be-done is tracing a claim that depends on connecting facts from two or more documents—for example, 'Company X acquired Company Y in 2022, and Company Y's CEO previously worked at Company Z'—and confirming that every link in that reasoning chain has a corresponding retrieved passage in the trace. Without this validation, a model can produce answers that sound coherent but contain unsupported logical leaps between sources.

Use this prompt after a trace has been captured, not during live generation. It assumes you have access to the full trace including the user query, all retrieval steps, the final generated output, and any intermediate reasoning or tool calls. The ideal user is someone debugging a specific production trace where a multi-hop answer raised suspicion—either through user feedback, an eval failure, or a spot check. You should have the trace open in an observability tool or exported as structured data before invoking this prompt. The prompt expects trace event IDs, retrieved document spans, and the generated output as inputs, and it will produce a step-by-step chain validation report with pass/fail status for each inferential hop.

This prompt is not a replacement for single-hop citation verification or for evaluating retrieval quality in isolation. If your answer only requires one source per claim, use the RAG Citation Verification Trace Prompt instead. If you are diagnosing why retrieval failed to return relevant documents, use the Evidence Gap Identification prompt. Do not use this prompt when the generated answer makes no cross-document connections, when the trace is incomplete or missing retrieval steps, or when you need real-time validation during generation. For high-risk domains such as healthcare, legal, or finance, always pair this prompt's output with human review of the flagged chain breaks—the prompt identifies where evidence is missing but cannot determine whether the missing evidence exists outside the retrieved set.

Before running this prompt, confirm that your trace contains: the user query, the list of retrieved documents with their content and trace event IDs, the final generated output, and any intermediate reasoning steps if your system exposes them. If your RAG system uses query rewriting or multi-step retrieval, include those intermediate queries and their results as well. The prompt will flag broken chains, missing intermediate evidence, and unsupported aggregations. After receiving the output, prioritize fixing chain breaks classified as 'critical'—these indicate claims presented as fact that have no evidential path in the trace. Use the trace event references in the output to navigate directly to the problematic retrieval or generation steps in your observability platform.

This prompt template is designed to be pasted directly into your trace analysis tool, evaluation pipeline, or LLM-as-judge harness. It accepts a generated claim, the full retrieval trace, and a set of constraints, then systematically verifies whether each inferential hop in a multi-hop reasoning chain is supported by the retrieved evidence. The template uses square-bracket placeholders for all dynamic inputs, making it straightforward to parameterize in code or a prompt management system.

textCopy
You are a trace analysis validator. Your task is to verify whether a multi-hop claim is fully supported by the retrieved evidence captured in a production trace.

## INPUT

**Claim to validate:**
[CLAIM]

**Retrieval trace (ordered by retrieval step):**
[RETRIEVAL_TRACE]

**Expected reasoning hops (the inferential steps the claim should require):**
[REASONING_HOPS]

**Source document metadata (if available):**
[SOURCE_METADATA]

## OUTPUT SCHEMA

Return a valid JSON object with this exact structure:
{
  "claim_summary": "string (brief restatement of the claim)",
  "hops": [
    {
      "hop_index": "integer",
      "hop_description": "string (what this step needs to establish)",
      "retrieved_evidence": "string or null (the passage that supports this hop, or null if none found)",
      "trace_event_id": "string or null (the trace event ID where this evidence was retrieved)",
      "support_status": "supported | partially_supported | unsupported | contradicted",
      "explanation": "string (why this status was assigned, referencing specific evidence or gaps)"
    }
  ],
  "chain_integrity": "intact | broken | incomplete",
  "broken_at_hop": "integer or null (the first hop index where the chain breaks, or null if intact)",
  "unsupported_aggregations": [
    {
      "description": "string (the aggregation or synthesis step that lacks evidence)",
      "missing_evidence_type": "string (what kind of evidence would be needed)"
    }
  ],
  "overall_grounding_score": "float between 0.0 and 1.0",
  "requires_human_review": "boolean",
  "review_reason": "string or null (why human review is recommended, or null if not required)"
}

## CONSTRAINTS

[CONSTRAINTS]

## INSTRUCTIONS

1. Decompose the claim into the reasoning hops provided in [REASONING_HOPS]. If no hops are provided, infer the likely inferential steps from the claim structure.
2. For each hop, search the [RETRIEVAL_TRACE] for evidence that directly supports the required inference. Do not assume evidence exists if it is not present in the trace.
3. If a hop combines information from multiple sources, verify that each source is present and that the combination is logically valid. Flag unsupported aggregations separately.
4. Mark the chain as broken if any hop is unsupported or contradicted, and identify the first failing hop.
5. Set requires_human_review to true if the claim involves safety, legal, financial, medical, or otherwise high-risk content, or if the grounding score is below [REVIEW_THRESHOLD].
6. Do not fabricate evidence. If the trace does not contain supporting passages, report the gap explicitly.
7. If [RISK_LEVEL] is high, apply stricter evidence requirements and lower the tolerance for partial support.

To adapt this template for your pipeline, replace each square-bracket placeholder with values from your trace store. The [RETRIEVAL_TRACE] should include the full sequence of retrieval calls, returned passages, and their trace event IDs. The [REASONING_HOPS] can be supplied by an upstream decomposition step or left empty to let the model infer hops from the claim. Set [REVIEW_THRESHOLD] to a value appropriate for your risk tolerance (0.7 is a common starting point for high-stakes domains). The [CONSTRAINTS] block can include domain-specific rules, such as requiring temporal alignment checks or prohibiting inference across documents from different time periods. Always run this prompt with structured output enforcement enabled and validate the returned JSON against the schema before ingesting results into your monitoring dashboard.

The Multi-Hop Citation Chain Validation Prompt is designed to operate as a post-hoc trace analysis step, not a real-time generation guard. It should be wired into your observability pipeline to consume structured trace data after a multi-hop RAG response has been generated and logged. The prompt expects a JSON payload containing the user's original query, the final generated answer, and the full retrieval chain—including each hop's query, retrieved documents, and the intermediate reasoning or synthesis steps captured in the trace. This is not a prompt you embed in the user-facing generation path; it belongs in an asynchronous evaluation worker, a nightly batch job, or a trace-review dashboard backend.

To integrate this prompt into an application, build a trace ingestion adapter that extracts the required fields from your observability store (e.g., LangSmith, Arize, or a custom trace database) and assembles them into the [TRACE_INPUT] placeholder. The adapter must handle variable-length retrieval chains—some traces will have two hops, others five or more—and normalize document metadata (source IDs, timestamps, relevance scores) into a consistent schema before passing them to the prompt. Implement a validation layer that checks the model's output JSON against the [OUTPUT_SCHEMA] you define: each hop in the chain must have a hop_index, claim, retrieved_evidence, support_status (supported, partially_supported, unsupported, missing_evidence), and chain_broken flag. If the model's output fails schema validation, retry once with the validation errors appended to the [CONSTRAINTS] field. After two failed retries, escalate the trace to a human review queue rather than silently accepting malformed output.

For high-stakes domains—legal, medical, financial—where a broken citation chain could cause real harm, add a human approval gate after the prompt runs. The system should surface traces where chain_broken is true or where any hop has support_status: unsupported to a review dashboard. The dashboard should display the original query, the generated answer, and a visual chain diagram with each hop color-coded by support status, allowing a human reviewer to confirm or override the model's assessment. Log every validation result, retry attempt, and human decision with the trace ID and prompt version for auditability. Model choice matters: use a model with strong structured output capabilities (GPT-4o, Claude 3.5 Sonnet) and set temperature=0 to minimize variance in chain analysis. Do not use this prompt with models under ~70B parameters for chains longer than three hops—smaller models tend to lose track of hop ordering and conflate evidence across retrieval steps.

Avoid wiring this prompt directly into a real-time user-facing pipeline. The latency of analyzing multi-hop chains—especially when retries are involved—will degrade user experience. Instead, run it asynchronously and use the results to trigger alerts, update dashboards, or feed into a continuous grounding quality metric. If you need real-time chain validation, consider a lightweight pre-check that only validates the final hop's citation alignment and defers full chain analysis to the async path. Finally, version your prompt template and track which version produced each validation result. When you update retrieval logic, embedding models, or the generation prompt, compare chain integrity scores across prompt versions to detect regressions before users notice.