Step-by-Step Correctness Grading Prompt Template

This prompt is for evaluation engineers and AI platform teams who need to automatically grade the correctness of reasoning traces produced by LLMs. It is designed for workflows where the model's step-by-step reasoning is as important as the final answer, such as in mathematical problem-solving, code debugging, medical diagnosis support, or legal analysis. Use this prompt when you need per-step correctness labels, cumulative error flags, and a final verdict that accounts for error propagation. The primary job-to-be-done is converting a raw reasoning trace and a ground-truth reference into a structured, machine-readable evaluation report that can trigger automated gates in a CI/CD pipeline or flag outputs for human review.

This prompt assumes you already have a reasoning trace to evaluate and a ground-truth reference or a set of verifiable facts. The ideal user is an evaluation lead or MLOps engineer integrating this prompt into an automated evaluation harness. Required context includes the full reasoning trace, the original problem statement, and a reference solution or fact set. The prompt is designed to be model-agnostic but performs best with models that have strong instruction-following and structured output capabilities. You should wire this into a system that can parse the JSON output and act on the final_verdict and error_propagation_detected fields, such as blocking a deployment or routing the trace for human annotation.

Do not use this prompt for grading creative writing, open-ended summarization quality, or tasks where there is no objectively correct reasoning path. It is also unsuitable for evaluating conversational quality, tone, or stylistic preferences. If your use case involves subjective quality dimensions, use a rubric-based evaluation prompt from the Rubric Design and Scoring group instead. For high-stakes domains like healthcare or legal analysis, the output of this prompt must be treated as a screening tool, not a final authority. Always route traces flagged with final_verdict: false or error_propagation_detected: true for human review before taking any action based on the model's output.

This section provides a copy-ready prompt template for the Step-by-Step Correctness Grading task. The template is designed to be wired directly into an evaluation harness. It expects a reasoning trace as input and produces per-step correctness labels, cumulative error flags, and a final verdict. All dynamic inputs are represented as square-bracket placeholders, which your application must resolve before sending the request to the model.

textCopy
You are an expert evaluation judge. Your task is to grade the correctness of a provided reasoning trace, step by step.

## INPUT
[REASONING_TRACE]

## CONTEXT
[GROUND_TRUTH_CONTEXT]

## OUTPUT_SCHEMA
You must respond with a valid JSON object conforming to this schema:
{
  "steps": [
    {
      "step_index": "integer",
      "step_text": "string",
      "correctness_label": "CORRECT | PARTIALLY_CORRECT | INCORRECT | UNVERIFIABLE",
      "error_type": "string | null",
      "justification": "string"
    }
  ],
  "cumulative_errors": [
    {
      "origin_step_index": "integer",
      "propagated_to_steps": ["integer"],
      "error_description": "string"
    }
  ],
  "final_verdict": "PASS | FAIL | UNCERTAIN",
  "verdict_rationale": "string"
}

## CONSTRAINTS
[CONSTRAINTS]

## EVALUATION_RUBRIC
[RUBRIC]

## INSTRUCTIONS
1. Parse the [REASONING_TRACE] into individual reasoning steps.
2. For each step, determine its correctness by comparing it against the [GROUND_TRUTH_CONTEXT] and the logical dependency on prior steps.
3. Assign a `correctness_label` to each step. Use `UNVERIFIABLE` if the [GROUND_TRUTH_CONTEXT] lacks the necessary information.
4. If a step is incorrect, identify the `error_type` from the allowed list in [CONSTRAINTS].
5. Track `cumulative_errors` where an error in one step causes a subsequent step to fail, even if that subsequent step's internal logic is sound.
6. Produce a `final_verdict` based on the [RUBRIC]. A single critical error that invalidates the conclusion should result in a `FAIL` verdict.
7. If the [REASONING_TRACE] is empty or cannot be parsed, return a `final_verdict` of `UNCERTAIN` with an appropriate `verdict_rationale`.

To adapt this template, replace the placeholders with concrete values in your application code. [REASONING_TRACE] should be the raw text output from the model under evaluation. [GROUND_TRUTH_CONTEXT] must contain the verified facts, source documents, or a reference answer against which steps are judged. [CONSTRAINTS] should be a list of allowed error types (e.g., Factual Error, Logical Fallacy, Calculation Error) and any domain-specific rules. [RUBRIC] defines the scoring criteria, such as 'A FAIL verdict requires at least one INCORRECT step that directly undermines the conclusion.' Before deploying, validate that the model's output strictly conforms to the OUTPUT_SCHEMA; a malformed JSON response should trigger a retry or fallback, not a silent failure in your evaluation pipeline.

This prompt is designed to operate as a stateless evaluation function inside a larger grading pipeline. It receives a reasoning trace and a reference solution, then returns structured per-step annotations. The harness should treat the LLM call as one component in a multi-stage workflow: pre-processing to normalize the trace format, the grading call itself, post-processing to validate the output schema, and a routing decision based on the final verdict and error flags. Do not use this prompt directly on user-facing chat interfaces; it belongs in batch evaluation jobs, CI/CD regression suites, or asynchronous review queues where latency tolerance is higher and output consistency matters more than speed.

Integration pattern: Wrap the prompt in a function with the signature grade_reasoning_trace(trace: str, reference: str, rubric: dict) -> GradingResult. Pre-process the input trace by splitting it into numbered steps if the model didn't produce them natively. Inject the step count into the [CONSTRAINTS] placeholder to bound the expected output array length. After the LLM call, validate the JSON response against a strict schema: require steps as an array of objects with step_index (int), correctness_label (enum of CORRECT, PARTIALLY_CORRECT, INCORRECT, NOT_EVALUABLE), error_type (nullable string), and justification (string). Reject and retry any response missing required fields or containing step indices outside the expected range. Log schema validation failures separately from content-level grading failures to distinguish model non-compliance from genuine reasoning errors.

Retry and fallback strategy: Implement up to two retries on schema validation failure, each time appending the validation error message to the [CONSTRAINTS] field. If the model still fails to produce valid JSON after two retries, escalate the entire trace to a human review queue and emit an EVALUATION_FAILED status rather than silently accepting a malformed result. For content-level concerns—such as a high density of INCORRECT labels or an error_propagation_detected flag set to true—route the trace to a secondary judge model for confirmation before surfacing it as a definitive failure. This two-judge pattern reduces false positives from a single LLM judge misreading ambiguous reasoning steps.

Model choice and tool use: This prompt benefits from models with strong instruction-following and structured output capabilities. Prefer models that support constrained JSON generation (e.g., GPT-4 with structured outputs, Claude with tool-use mode) to reduce schema violations at the source. If your model provider supports it, define the output as a tool call with a typed schema rather than relying solely on the prompt's [OUTPUT_SCHEMA] description. Do not give the grading model access to external tools, retrieval, or code execution during evaluation—the grader should judge the trace as presented, not attempt to solve the problem itself. For traces involving mathematical reasoning, consider a pre-grading verification step that runs calculations through a deterministic arithmetic validator before the LLM judge evaluates the reasoning logic, separating computational correctness from logical correctness.

Human review integration: Set a threshold on the final_verdict and cumulative error count to trigger human review. For high-stakes domains (healthcare, legal, finance), any trace with one or more INCORRECT steps or an error_propagation_detected flag should route to a human evaluator before the grade is finalized. Store the LLM judge's annotations alongside the human reviewer's corrections to build a calibration dataset for future judge alignment. Avoid using this prompt as the sole arbiter in production systems where grading errors could cause downstream harm—treat it as a filter that reduces human review load, not a replacement for expert judgment.