Decomposition Strategy Quality Evaluation Prompt

This prompt is designed for AI evaluation engineers and agent platform teams who need to programmatically grade the quality of a model's problem decomposition strategy. The core job-to-be-done is turning a subjective assessment of 'did the model break this problem down well?' into a structured, repeatable score. You should use this when you are evaluating reasoning traces from agent planning, chain-of-thought outputs, or task decomposition modules, and you need to verify that the sub-problems are complete, logically independent, correctly ordered, and broken down to an appropriate level of granularity before the agent begins execution.

The ideal user has a batch of reasoning traces and a reference definition of the original complex problem. You must provide the full reasoning trace and the original problem statement as inputs. This prompt is not a general-purpose quality rater; it is specifically calibrated for decomposition strategy. Do not use it to grade final answer correctness, factual accuracy, or stylistic quality. It is also a poor fit for single-step tasks where no decomposition occurs, or for evaluating creative brainstorming where divergent, overlapping ideas are a feature rather than a bug. If your agent's plan is already executing and you want to evaluate tool-use correctness, switch to a tool-use evaluation prompt instead.

Before wiring this into an automated pipeline, ensure you have a human-annotated calibration set to validate the LLM judge's scores against your team's standards. The output is a structured scorecard, not a binary pass/fail, so downstream systems should implement threshold logic based on your risk tolerance. For high-stakes agent workflows—such as those in healthcare, finance, or safety-critical operations—always route low decomposition scores for human review before allowing autonomous execution to proceed.

This prompt template evaluates how well a reasoning trace decomposes a complex problem into sub-problems. It produces structured scores for decomposition completeness, sub-problem independence, ordering logic, and granularity appropriateness. The template uses square-bracket placeholders so you can swap in your own reasoning trace, evaluation rubric, output schema, and constraints without rewriting the core evaluation logic.

textCopy
You are an expert evaluation judge specializing in problem decomposition quality. Your task is to evaluate how well a reasoning trace breaks down a complex problem into sub-problems.

## INPUT

**Original Problem:**
[ORIGINAL_PROBLEM]

**Reasoning Trace to Evaluate:**
[REASONING_TRACE]

## EVALUATION DIMENSIONS

Evaluate the decomposition strategy across these four dimensions using the provided scoring scale:

### 1. Decomposition Completeness (1-5)
- Does the trace identify all necessary sub-problems required to solve the original problem?
- Are there missing sub-problems that would prevent a complete solution?
- Score 5: All required sub-problems identified. Score 1: Critical sub-problems missing.

### 2. Sub-Problem Independence (1-5)
- Are sub-problems defined so they can be solved independently where possible?
- Does the trace avoid unnecessary coupling between sub-problems?
- Score 5: Sub-problems are cleanly separated with minimal overlap. Score 1: Sub-problems are entangled and interdependent.

### 3. Ordering Logic (1-5)
- Are sub-problems sequenced in a logical order where dependencies are respected?
- Does the trace explain why certain sub-problems must precede others?
- Score 5: Clear, justified ordering with explicit dependency reasoning. Score 1: Arbitrary or incorrect ordering that blocks progress.

### 4. Granularity Appropriateness (1-5)
- Are sub-problems broken down to an appropriate level of detail?
- Does the trace avoid over-decomposition (too fine-grained) or under-decomposition (too coarse)?
- Score 5: Each sub-problem is actionable and appropriately scoped. Score 1: Sub-problems are either trivial or impossibly broad.

## OUTPUT SCHEMA

Return a JSON object with this exact structure:
{
  "overall_score": <float between 1.0 and 5.0>,
  "dimension_scores": {
    "decomposition_completeness": <int 1-5>,
    "sub_problem_independence": <int 1-5>,
    "ordering_logic": <int 1-5>,
    "granularity_appropriateness": <int 1-5>
  },
  "strengths": [<list of specific strengths observed>],
  "weaknesses": [<list of specific gaps or issues>],
  "missing_sub_problems": [<list of sub-problems the trace should have included but didn't>],
  "ordering_issues": [<list of sequencing problems with step references>],
  "granularity_issues": [<list of over-decomposition or under-decomposition examples>],
  "overall_assessment": "<2-3 sentence summary of decomposition quality>"
}

## CONSTRAINTS

[CONSTRAINTS]

## EVALUATION INSTRUCTIONS

1. First, extract all sub-problems explicitly stated in the reasoning trace.
2. Compare against what a complete decomposition should include for this problem type.
3. Check whether sub-problems overlap or depend on each other unnecessarily.
4. Verify that the ordering respects actual dependencies.
5. Assess whether each sub-problem is scoped appropriately for solution.
6. Provide specific evidence from the trace for every score and finding.
7. If the trace is too short to evaluate decomposition, note this and score accordingly.

## EXAMPLES

[EXAMPLES]

Adaptation guidance: Replace [ORIGINAL_PROBLEM] with the problem statement the model was given. Replace [REASONING_TRACE] with the full reasoning output you want to evaluate. Use [CONSTRAINTS] to add domain-specific rules such as required sub-problem categories, minimum sub-problem counts, or forbidden decomposition patterns. Use [EXAMPLES] to provide scored examples that calibrate the judge to your quality standards. If you need a different output shape, modify the JSON schema block directly. For high-stakes evaluation pipelines, add a confidence field to the output schema and require the judge to flag low-confidence assessments for human review.

This prompt is designed to run inside an automated evaluation harness, not as a one-off manual review tool. The typical integration point is after an agent or reasoning model has produced a decomposition trace. You feed the original problem statement and the decomposition trace into this prompt, and the LLM judge returns a structured JSON scorecard. The harness should treat this as a model-graded evaluation step that can be batched, logged, and compared across prompt versions or model changes.

Validation and retry logic is critical because the output schema includes numeric scores, enum labels, and arrays of finding objects. Implement a JSON schema validator that checks: all required fields are present, overall_score is an integer between 1 and 5, each dimension_score is within its defined range, and critical_flaws contains valid enum values like missing_subproblem or dependency_cycle. If validation fails, retry up to two times with the same prompt but append the validation error message as a [CONSTRAINTS] note. After three failures, flag the trace for human review rather than silently accepting a malformed score. Log every validation failure and retry attempt for later judge calibration analysis.

Model choice matters. This prompt requires strong reasoning capabilities to assess decomposition logic. Use a capable model like GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro as the judge. Avoid using smaller or faster models for this evaluation task because they tend to miss subtle dependency violations and produce inflated scores. If cost is a concern, consider sampling a subset of traces for full evaluation rather than downgrading the judge model. Human review gates should be triggered when the overall score is 2 or below, when critical flaws are detected, or when the judge's confidence field (if you add one) falls below a threshold. Route flagged traces to a review queue where a human can confirm or override the automated score before the decomposition strategy is used in production.

Integration pattern: Wire this prompt as a post-processing step in your agent evaluation pipeline. After an agent produces a decomposition, store the raw trace. Then call this evaluation prompt asynchronously, store the scorecard alongside the trace, and update your evaluation dashboard. If you're running regression tests across prompt versions, compare the distribution of decomposition scores to detect regressions. A drop in sub_problem_independence or ordering_logic scores often signals that a planning prompt change introduced subtle breakage. Finally, periodically run inter-rater reliability checks by having a second LLM judge or a human evaluator score the same traces and comparing agreement rates on each dimension.