Rubric for Self-Correction Success Rate Prompt Template

This prompt is for AI engineers and evaluation leads who need to measure whether self-correction attempts actually improve output quality. Use it when you have a retry or repair loop in production and need a repeatable, model-graded rubric to score each correction attempt. The rubric grades whether errors were fixed, whether new errors were introduced, and whether the output degraded. It also defines stopping criteria so your system knows when to stop retrying and escalate. The core job-to-be-done is turning an opaque, iterative repair process into a measurable, automated quality gate that prevents infinite retry loops and silent degradation.

Do not use this prompt for single-pass output evaluation where no correction history exists. It assumes you have the original output, the error or feedback that triggered correction, and the corrected output available for comparison. This prompt is also inappropriate for workflows where the cost of a bad correction is low and a simple retry without evaluation is sufficient. It is designed for high-stakes pipelines—such as code generation, structured data extraction, or compliance document drafting—where each correction attempt must be audited for regression. Before using this prompt, ensure your system logs the full correction chain: the initial output, the validator or human feedback that triggered the repair, and every subsequent correction attempt. Without this trace, the rubric has nothing to grade.

Once you have confirmed that your system captures correction history, use this prompt to generate a rubric that scores each attempt on fix completeness, regression introduction, and quality delta. The resulting rubric should be wired into your retry controller so that the system can decide whether to accept a correction, retry with modified instructions, or escalate to a human. Avoid using this prompt if your retry loop is purely syntactic—such as JSON repair—where deterministic validators are faster and cheaper than an LLM judge.

This prompt template instructs an LLM judge to evaluate a sequence of self-correction attempts against a defined rubric. It grades whether each correction fixes the original error, introduces new errors, or degrades output quality. The template is designed to be wired into an automated evaluation harness that tracks correction attempt quality and determines when retries should stop.

textCopy
You are an evaluation judge grading self-correction attempts. Your task is to assess whether a correction attempt successfully resolves a previously identified error without introducing new problems.

## INPUT DATA
- Original Output: [ORIGINAL_OUTPUT]
- Error Description: [ERROR_DESCRIPTION]
- Correction Attempt: [CORRECTION_ATTEMPT]
- Correction Attempt Number: [ATTEMPT_NUMBER]
- Maximum Allowed Attempts: [MAX_ATTEMPTS]
- Domain Context: [DOMAIN_CONTEXT]

## EVALUATION RUBRIC
Score each correction attempt on the following dimensions using a 1-5 scale where 1 is worst and 5 is best:

### 1. Error Resolution (1-5)
- 5: Original error is completely resolved with no residual issues
- 4: Original error is resolved but minor stylistic or formatting differences remain
- 3: Original error is partially resolved; some aspects of the error persist
- 2: Original error is mostly unresolved; only superficial changes made
- 1: Original error is not resolved at all or made worse

### 2. Error Introduction (1-5, inverted scale)
- 5: No new errors introduced; output quality is maintained or improved
- 4: Minor new issues introduced that don't affect core correctness
- 3: One moderate new error introduced that affects non-critical aspects
- 2: Multiple new errors introduced or one significant error
- 1: Correction introduces critical errors worse than the original problem

### 3. Quality Preservation (1-5)
- 5: All original correct content preserved; only error portions changed
- 4: Most correct content preserved; minimal unnecessary changes
- 3: Some correct content unnecessarily modified but intent preserved
- 2: Significant correct content altered or removed unnecessarily
- 1: Output completely rewritten; original correct content lost

### 4. Correction Efficiency (1-5)
- 5: Minimal, targeted change that precisely addresses the error
- 4: Slightly broader change than necessary but still focused
- 3: Moderate scope change; some unnecessary modifications included
- 2: Overly broad correction attempt; significant unrelated changes
- 1: Complete rewrite or irrelevant changes that don't target the error

## STOPPING CRITERIA ASSESSMENT
Based on the scores above, determine if correction should continue:
- CONTINUE: Scores indicate progress; further attempts may succeed
- STOP_SUCCESS: Error resolved (Error Resolution >= 4, Error Introduction >= 4)
- STOP_DEGRADING: Quality is declining (any score <= 2, or scores decreasing across attempts)
- STOP_STALLED: No meaningful progress after [STALL_THRESHOLD] attempts
- STOP_MAX_ATTEMPTS: Maximum attempts reached without resolution

## OUTPUT FORMAT
Return a JSON object with this exact structure:
{
  "attempt_number": integer,
  "scores": {
    "error_resolution": integer (1-5),
    "error_introduction": integer (1-5),
    "quality_preservation": integer (1-5),
    "correction_efficiency": integer (1-5)
  },
  "composite_score": float (average of four dimensions),
  "stopping_decision": "CONTINUE" | "STOP_SUCCESS" | "STOP_DEGRADING" | "STOP_STALLED" | "STOP_MAX_ATTEMPTS",
  "justification": "Brief explanation of scores and stopping decision",
  "new_errors_detected": ["List of new errors introduced, if any"],
  "correction_quality_trend": "IMPROVING" | "STABLE" | "DEGRADING" | "FIRST_ATTEMPT"
}

## CONSTRAINTS
- Score only the correction attempt, not the original output in isolation
- If this is attempt 1, mark correction_quality_trend as "FIRST_ATTEMPT"
- For attempt 2+, compare against previous attempt scores to determine trend
- Flag any safety, compliance, or factual regression as critical (score <= 2 on error_introduction)
- If domain context is provided, apply domain-specific quality expectations

To adapt this template, replace each square-bracket placeholder with your actual data. The [DOMAIN_CONTEXT] field is critical for specialized domains like healthcare or legal where error severity definitions differ. For production use, wire this prompt into a retry loop that feeds the judge's stopping_decision back to your orchestrator. Always log the full score object for later analysis of correction attempt quality trends across your system.

This rubric is designed to operate as a post-hoc judge inside an automated evaluation pipeline, not as a real-time guardrail. The typical harness runs after a self-correction loop completes: the original output, the correction attempt, and any error signals or validator feedback are packaged into the [CORRECTION_LOG] input. The LLM judge then scores the correction attempt across the defined dimensions and returns a structured JSON verdict. Because this rubric evaluates a meta-process (the repair attempt itself), the harness must preserve the full chain of evidence—original failure, correction action, and final output—rather than just comparing two static texts.

Pipeline integration follows a standard eval loop pattern. After each self-correction attempt, serialize the attempt into a structured log containing: the original prompt, the initial output, the validator error or failure signal that triggered correction, the correction prompt used, and the final corrected output. Feed this log into the rubric prompt as [CORRECTION_LOG]. The judge returns a JSON object with dimension scores, an overall success classification (successful_correction, partial_fix, degraded_output, new_errors_introduced), and a stopping recommendation. Validation layer: parse the JSON response and validate that all required fields are present, scores fall within the defined scale, and the success classification matches one of the allowed enum values. If validation fails, retry once with a stripped-down prompt that asks only for the missing fields. After two failures, flag the evaluation record for human review rather than silently accepting a malformed score.

Model choice matters for this rubric. The judge must reason about whether a correction fixed the original problem without introducing new errors—a task that requires comparing three states (before, during, after) and detecting subtle regressions. Use a capable model (GPT-4o, Claude 3.5 Sonnet, or equivalent) for the judge role. Avoid using the same model that performed the self-correction as its own judge; self-evaluation bias inflates success rates. Logging and traceability: store the full correction log, the judge's raw response, the parsed scores, and any validation errors in your evaluation database. Tag each record with the prompt version, model IDs for both the worker and judge, and a unique correction attempt ID. This trace is essential for debugging when the rubric reports high success rates but human reviewers disagree.

Human review gates should trigger when: (a) the judge returns degraded_output or new_errors_introduced, (b) the overall success score falls below your calibrated threshold (start at 0.7 on a 0-1 scale and adjust based on your risk tolerance), or (c) the judge's confidence field (if you include one) drops below 0.8. For high-stakes domains like healthcare, legal, or finance, route all partial_fix classifications to human review as well. Calibration step: before trusting this rubric in production, run at least 50 correction attempts through both the LLM judge and a human evaluator. Compute agreement rates per dimension and adjust the rubric's anchor descriptions until Cohen's kappa exceeds 0.7. Recalibrate whenever you change the worker model, the correction strategy, or the underlying task domain.

What to avoid: Don't use this rubric to evaluate corrections in real-time user-facing loops—the judge adds latency and cost that belongs in offline evaluation. Don't treat the judge's scores as ground truth without periodic human spot-checks; judge drift is real, especially as worker model behavior changes. Don't aggregate success rates across different task types; a correction that works for format errors may fail for factual errors, and averaging hides this. Instead, segment your evaluation results by error category (format, factual, instruction-following, tool-use) and track correction success rates separately. Wire these segmented metrics into your eval dashboard with alerts when any category's success rate drops below your baseline.