Decomposition Quality Evaluation Prompt

This prompt is for prompt engineers and QA teams who need an automated, model-graded evaluation of sub-question quality before those sub-questions enter a retrieval pipeline. Use it when you have a decomposition step that breaks a complex user query into smaller, independently answerable sub-questions and you need a production gate to catch incomplete, overlapping, or irrelevant decompositions. The prompt acts as an LLM judge, scoring the generated sub-questions on completeness, independence, and relevance against the original query. It is designed for RAG systems, agentic research workflows, and any multi-step retrieval architecture where bad sub-questions cause downstream answer failure.

The ideal deployment point is immediately after a decomposition prompt runs and before any retrieval calls are made. Wire the judge as a synchronous validation step: take the original user query and the list of candidate sub-questions, pass them through this evaluation prompt, and check the pass/fail score against your threshold. If the decomposition fails, you can trigger a retry with the decomposition prompt, fall back to a simpler single-query retrieval, or escalate for human review. For high-throughput systems, consider sampling evaluations rather than judging every decomposition, and log all scores to track decomposition quality drift over time.

Do not use this prompt for evaluating final answers, retrieval quality, or single-hop query rewrites. It assumes you already have a set of candidate sub-questions to evaluate. If your system generates only one rewritten query rather than multiple sub-questions, use a query validation prompt instead. For regulated or high-risk domains such as healthcare or legal workflows, the judge score should inform but not replace human review of the decomposition before retrieval executes. The rubric is designed for production gating, not for academic benchmarking, so calibrate the pass threshold against your own failure tolerance and observed retrieval outcomes.

This prompt template acts as an automated evaluator for your query decomposition pipeline. It takes the user's original complex question and the set of sub-questions your system generated, then scores each sub-question against a defined rubric. The primary job is to gate production deployments: if the decomposition quality falls below a threshold, the system should fall back to a simpler retrieval strategy or escalate for human review rather than serving a bad multi-hop answer.

textCopy
You are a strict evaluator of query decomposition quality. Your task is to score a set of generated sub-questions against the original user query.

# INPUT
[ORIGINAL_QUERY]: The user's complex, multi-part question.
[SUB_QUESTIONS]: A JSON array of generated sub-question objects. Each object has an "id" (string) and "question" (string) field.

# EVALUATION RUBRIC
For each sub-question, assign a score of 1 (pass) or 0 (fail) for each of the following criteria:

1. **Completeness**: Does the set of sub-questions, taken together, cover all distinct parts of the original query? No part of the original intent should be missing.
2. **Independence**: Can each sub-question be answered without requiring the answer to another sub-question first? Flag any sub-question that depends on the result of a sibling sub-question.
3. **Relevance**: Is every sub-question directly relevant to answering the original query? Flag any sub-question that introduces unrelated topics or unnecessary tangents.
4. **Granularity**: Are the sub-questions broken down to the right level of detail? They should be specific enough to retrieve focused evidence but not so granular that they become trivial keyword matches.

# OUTPUT SCHEMA
Return ONLY a valid JSON object with the following structure. Do not include any text outside the JSON.

{
  "overall_pass": boolean, // true only if ALL sub-questions pass ALL criteria
  "sub_question_evaluations": [
    {
      "id": "string",
      "completeness": { "score": 0 or 1, "reasoning": "string" },
      "independence": { "score": 0 or 1, "reasoning": "string" },
      "relevance": { "score": 0 or 1, "reasoning": "string" },
      "granularity": { "score": 0 or 1, "reasoning": "string" }
    }
  ],
  "missing_coverage": ["string"], // List any parts of the original query not covered by any sub-question
  "critical_failure": "string or null" // If a fatal flaw exists, describe it here; otherwise null
}

# CONSTRAINTS
- Do not answer the sub-questions. Only evaluate their quality as decomposition artifacts.
- If the original query is ambiguous, note it in `critical_failure` but still evaluate the sub-questions as provided.
- Be strict. A score of 1 means the criterion is fully satisfied with no significant issues.

To adapt this template, replace [ORIGINAL_QUERY] with the user's raw input and [SUB_QUESTIONS] with the JSON output from your decomposition prompt. The overall_pass boolean provides a simple gating signal: if false, block the multi-hop retrieval and either re-prompt for better decomposition or fall back to a single-shot RAG call. The missing_coverage array is particularly valuable for debugging—it tells you exactly which parts of the user's intent your decomposition prompt is systematically dropping. Store the full evaluation JSON in your trace logs to track decomposition quality over time and detect prompt drift.

This prompt functions as an LLM judge and should be treated as a gating step in your query decomposition pipeline, not a one-off manual review. After your primary decomposition prompt generates a set of sub-questions, route the original query and the generated sub-questions to this evaluator before any retrieval is executed. The evaluator returns a structured score and a pass/fail decision, which your application harness must read to decide whether to proceed with retrieval, regenerate the sub-questions, or escalate for human review. The core integration point is a conditional branch: if pass is false or the completeness_score falls below your threshold (typically 0.8), do not waste compute on retrieval with broken sub-questions.

Wire the prompt into an async evaluation step with explicit retry logic. On a fail verdict, capture the failure_reasons array from the output and feed it back as context into a retry of your decomposition prompt, instructing the model to address the specific gaps identified (e.g., 'The previous decomposition missed the temporal constraint; regenerate sub-questions covering the Q3 2024 time range'). Implement a maximum retry loop of 2 attempts to prevent infinite regeneration cycles. If the evaluator still returns a fail after retries, escalate the original query to a human review queue with the failed sub-questions and failure reasons attached. Log every evaluation result—including scores, failure reasons, and the final disposition—as structured metadata on the query trace for observability. For model choice, use a fast, cost-effective model (such as GPT-4o-mini or Claude Haiku) for this evaluation step, as it is a classification and scoring task that does not require a frontier model. The evaluator should run with temperature=0 to ensure deterministic, repeatable judgments.

Before deploying, build a regression test harness around this evaluator. Create a golden dataset of 20–30 query-and-sub-question pairs with known expected verdicts: include cases that should pass (complete, independent, relevant sub-questions), cases that should fail due to missing coverage, cases with dependent sub-questions that cannot run independently, and edge cases with ambiguous queries. Run the evaluator against this dataset on every prompt change and assert that the pass/fail accuracy against your labels exceeds 90%. Monitor production drift by sampling evaluation verdicts weekly and checking for unexpected pass rate changes. Do not use this evaluator as the sole quality signal for user-facing answers—it gates decomposition quality, not final answer correctness. The downstream RAG answer must still pass through its own groundedness and factuality checks.