Decomposed Reasoning Evaluation Rubric Prompt

This prompt is for AI engineers and QA teams who need an automated, repeatable way to evaluate the quality of decomposed reasoning chains produced by other LLM prompts. The job-to-be-done is regression testing: you have a multi-hop RAG system or a reasoning agent that breaks complex questions into sub-questions and evidence chains, and you need to know if a change to your reasoning prompt made the outputs better or worse. The ideal user is someone running an evaluation harness who already has a golden dataset of questions, retrieved contexts, and reference reasoning chains. You should use this prompt when you need a structured scorecard that assesses logical coherence, evidence grounding, step necessity, and conclusion validity, not just a vague 'looks good' judgment.

Do not use this prompt as a real-time guardrail in a production request path. It is designed for offline evaluation and CI/CD pipelines where latency and cost per judgment are acceptable. It is also not a substitute for human review when the reasoning chain informs high-stakes decisions in regulated domains—use it to filter and prioritize human review, not to replace it. The prompt requires a detailed rubric, the reasoning chain to be evaluated, the source evidence used, and the original question as inputs. Without these, the evaluation will be unreliable. The output is a structured scorecard with dimension-level scores and an overall pass/fail recommendation, which you can log, trend over time, and use as a release gate.

Before wiring this into your eval harness, define your minimum acceptable scores per dimension and decide whether a single failing dimension fails the whole chain. Start by running this prompt against a set of known-good and known-bad reasoning chains to calibrate thresholds. Avoid the temptation to tweak the rubric until everything passes—use the failures to find real bugs in your reasoning prompts. If you observe inconsistent grading across similar chains, add more detail to the rubric's dimension descriptions or include few-shot examples of scored chains to anchor the LLM judge.

This prompt template is designed to be used inside an evaluation harness. It instructs a judge model to assess the quality of a reasoning chain—typically generated by a multi-hop RAG system—against a detailed rubric. The output is a structured scorecard that can be parsed and logged by your CI/CD pipeline, enabling automated regression testing of your reasoning prompts. The template uses square-bracket placeholders for all dynamic inputs, making it safe to copy directly into your codebase.

textCopy
You are an expert evaluator grading the quality of a decomposed reasoning chain. Your task is to assess the provided reasoning chain against a detailed rubric and produce a structured scorecard.

## INPUTS

**User Question:**
[USER_QUESTION]

**Retrieved Context (multiple documents may be provided):**
[RETRIEVED_CONTEXT]

**Reasoning Chain to Evaluate:**
[REASONING_CHAIN]

## RUBRIC

Evaluate the reasoning chain on the following dimensions. For each dimension, assign a score from 1 (worst) to 5 (best) and provide a brief, evidence-based justification.

1. **Logical Coherence:** Does the reasoning flow logically from one step to the next? Are there any logical leaps, circular reasoning, or contradictions?
2. **Evidence Grounding:** Is every factual claim in the reasoning chain explicitly supported by the provided context? Flag any unsupported claims.
3. **Step Necessity:** Is each step in the chain necessary to reach the conclusion? Are there redundant or irrelevant steps?
4. **Conclusion Validity:** Does the final conclusion follow necessarily from the reasoning steps and the evidence? Is it a valid inference?
5. **Source Fidelity:** Are citations and references to the source documents accurate? Does the chain misattribute or fabricate source details?

## OUTPUT SCHEMA

Return ONLY a valid JSON object conforming to this schema:
{
  "overall_score": <float between 1.0 and 5.0>,
  "overall_verdict": "PASS" | "FAIL" | "BORDERLINE",
  "dimensions": [
    {
      "name": "Logical Coherence",
      "score": <integer 1-5>,
      "justification": "<string>"
    },
    {
      "name": "Evidence Grounding",
      "score": <integer 1-5>,
      "justification": "<string>"
    },
    {
      "name": "Step Necessity",
      "score": <integer 1-5>,
      "justification": "<string>"
    },
    {
      "name": "Conclusion Validity",
      "score": <integer 1-5>,
      "justification": "<string>"
    },
    {
      "name": "Source Fidelity",
      "score": <integer 1-5>,
      "justification": "<string>"
    }
  ],
  "critical_flags": ["<string describing any critical failure, e.g., hallucinated citation>"]
}

## CONSTRAINTS

- Do not add commentary outside the JSON object.
- If the reasoning chain is empty or nonsensical, set overall_score to 1.0 and overall_verdict to "FAIL".
- If any dimension receives a score of 1 or 2, overall_verdict must be "FAIL" or "BORDERLINE".
- Be strict about evidence grounding: if a claim is not in the context, flag it.

To adapt this template, replace the placeholders with your actual data. [USER_QUESTION] is the original complex query. [RETRIEVED_CONTEXT] should contain the full text of all documents retrieved across all hops, clearly delimited. [REASONING_CHAIN] is the output from your multi-hop reasoning prompt that you want to evaluate. The rubric dimensions can be adjusted to match your specific quality criteria—for example, you might add 'Temporal Consistency' for event-sequencing tasks. The output schema is designed for direct ingestion by an automated test runner; ensure your harness parses the JSON and logs the scorecard alongside the prompt version and input data. For high-stakes domains, always include a human review step for any verdict of 'BORDERLINE' or 'FAIL' before blocking a release.

This prompt is designed to be used as an LLM-as-judge within an automated evaluation pipeline, not as a one-off manual grading tool. The primary integration point is a CI/CD or pre-release testing workflow that runs whenever the upstream reasoning prompt (the prompt being evaluated) is modified. The harness should retrieve a golden dataset of complex questions, their corresponding retrieved context, and the reasoning chains generated by the candidate prompt. Each reasoning chain is then fed into this rubric prompt alongside the original question and source evidence to produce a structured scorecard.

For a robust implementation, wrap the prompt call in a validation and retry layer. The expected output is a JSON object matching a strict schema (fields like logical_coherence_score, evidence_grounding_score, overall_verdict, and an array of critical_failures). After the LLM call, validate the JSON structure. If parsing fails or required fields are missing, implement a retry loop (maximum 2-3 attempts) that feeds the raw output and the validation error back to the model for self-correction. Log every raw response and parsing failure to an observability store for later analysis. For model choice, a capable frontier model with strong instruction-following (like GPT-4o or Claude 3.5 Sonnet) is recommended for the judge role, as it must apply a detailed rubric consistently. Avoid using a smaller, cheaper model for the judge unless you have calibrated its grading against a human-annotated baseline.

The final step in the harness is aggregation and gating. Collect the scorecards across the entire golden dataset. Calculate the pass rate based on your predefined overall_verdict threshold (e.g., 80% of test cases must pass). If the pass rate falls below the threshold, the CI pipeline should fail, blocking the reasoning prompt change from being released. Store all scorecards as artifacts of the test run. Crucially, flag any test case where the critical_failures array is non-empty for immediate human review, as these represent logical leaps or hallucinations that the judge deemed severe. This harness transforms a subjective quality assessment into a repeatable, automated regression gate.