User Correction Incorporation Evaluation Prompt

This prompt is designed for support automation teams and chat product engineers who need to measure whether an AI assistant correctly incorporates user corrections during a conversation. The core job-to-be-done is automated quality assurance: you have a chat log where a user corrected the assistant (e.g., 'No, my account number is 8842, not 8824'), and you need a structured, per-correction score indicating whether subsequent assistant turns reflect the correction or silently revert to the original error. This is not a prompt for generating responses; it is an evaluation prompt that acts as a specialized LLM judge.

Use this prompt when you have identified correction events in a transcript and need to score the assistant's response across multiple subsequent turns. It requires three concrete inputs: the full conversation transcript, the specific turn index where the correction occurred, and the exact correction statement made by the user. The prompt is most valuable in regression testing pipelines, pre-release chat quality gates, and production monitoring dashboards where you need to track a specific failure mode—correction reversion—that generic conversation quality scores often miss. Do not use this prompt for single-turn evaluations, for scoring corrections where the user didn't explicitly state the correct information, or for conversations without clear correction events.

Before implementing, ensure your pipeline can extract correction events from transcripts (either via another LLM call or pattern matching). The prompt assumes you already know where corrections happened. For high-stakes domains like healthcare or finance, pair this automated score with human review of flagged reversions. The output is a structured JSON scorecard, not a free-text critique, making it directly ingestible by your metrics dashboard or CI/CD pipeline. If you need to evaluate broader conversation quality across multiple dimensions, use the Multi-Turn Conversation Scorecard Generation Prompt instead.

This prompt template is designed to be dropped into an evaluation harness that processes conversation logs containing at least one explicit user correction. The template instructs the model to act as an evaluator, identify correction events, and score the assistant's subsequent response on acknowledgment, incorporation, and persistence. Use this template as the core instruction block for an LLM judge, and adapt the placeholders to match your specific conversation format, scoring rubric, and output requirements.

textCopy
You are an evaluator grading an AI assistant's ability to incorporate user corrections during a multi-turn conversation. Your task is to identify every instance where the user explicitly corrects the assistant or a piece of information, and then evaluate how the assistant handles that correction in its immediately following response and all subsequent turns.

## INPUT DATA

[CONVERSATION_LOG]

## CORRECTION IDENTIFICATION

First, extract every user turn that contains an explicit correction. A correction is a statement that directly contradicts, amends, or retracts information previously provided by the user or the assistant. Do not flag clarifications, elaborations, or new topic shifts as corrections.

For each correction event, record:
- `correction_id`: A unique identifier (e.g., "C1", "C2")
- `correction_turn_index`: The turn number where the correction occurs
- `corrected_claim`: The original claim or information being corrected
- `correction_content`: The new, corrected information provided by the user
- `correction_type`: One of ["factual_correction", "preference_change", "constraint_update", "retraction", "instruction_override"]

## EVALUATION CRITERIA

For each correction event, evaluate the assistant's response in the turn immediately following the correction AND all subsequent turns up to the end of the conversation. Score each criterion on a scale of 1-5, where 1 is complete failure and 5 is perfect handling.

### 1. Acknowledgment (1-5)
Did the assistant explicitly acknowledge the correction? Look for phrases that confirm understanding of the change.
- 5: Clear, explicit acknowledgment that names the corrected item
- 3: Implicit acknowledgment through behavior change without verbal confirmation
- 1: No acknowledgment; assistant continues as if correction never happened

### 2. Incorporation Accuracy (1-5)
Did the assistant's subsequent responses reflect the corrected information accurately?
- 5: All subsequent responses fully incorporate the correction with no errors
- 3: Partial incorporation with minor inconsistencies
- 1: Assistant reverts to the original incorrect information or introduces new errors

### 3. Persistence (1-5)
Did the corrected information persist through all remaining turns, or did the assistant revert to the original error later?
- 5: Correction is maintained consistently through all remaining turns
- 3: Correction is maintained in the immediate next turn but drifts later
- 1: Correction is lost immediately or within one additional turn

### 4. No Over-Correction (1-5)
Did the assistant avoid over-applying the correction to unrelated parts of the conversation?
- 5: Correction is applied only where relevant; no collateral damage
- 3: Minor over-application that doesn't derail the conversation
- 1: Assistant incorrectly applies the correction to unrelated topics or constraints

## OUTPUT FORMAT

Return a JSON object with the following structure:

{
  "conversation_id": "string",
  "total_turns": number,
  "correction_events": [
    {
      "correction_id": "string",
      "correction_turn_index": number,
      "corrected_claim": "string",
      "correction_content": "string",
      "correction_type": "string",
      "scores": {
        "acknowledgment": number,
        "incorporation_accuracy": number,
        "persistence": number,
        "no_over_correction": number
      },
      "composite_score": number,
      "evidence": {
        "acknowledgment_quote": "string or null",
        "incorporation_check_turns": [number],
        "reversion_detected": boolean,
        "reversion_turn_index": number or null,
        "over_correction_detected": boolean,
        "over_correction_detail": "string or null"
      },
      "failure_mode": "string or null"
    }
  ],
  "aggregate_metrics": {
    "correction_count": number,
    "average_composite_score": number,
    "corrections_fully_incorporated": number,
    "corrections_with_reversion": number,
    "worst_performing_criterion": "string"
  },
  "overall_assessment": "string"
}

## CONSTRAINTS

[CONSTRAINTS]

## EVALUATION NOTES

- If the user issues multiple corrections in a single turn, treat each as a separate correction event.
- If the assistant's response to a correction is cut off or the conversation ends before a response, mark the correction as unevaluable and exclude it from aggregate scores.
- If the assistant correctly incorporates a correction but later the user provides new information that supersedes it, this is NOT a reversion. Only flag reversion when the assistant returns to the original error unprompted.
- For conversations with zero corrections, return an empty `correction_events` array and note this in the `overall_assessment`.

To adapt this template for your evaluation pipeline, replace [CONVERSATION_LOG] with your actual conversation data formatted as turn-by-turn JSON or a structured text representation. The [CONSTRAINTS] placeholder should be populated with any domain-specific rules, such as minimum score thresholds for pass/fail decisions, special handling for PII in conversation logs, or additional criteria like tone appropriateness. If your use case involves regulated domains, add a constraint requiring human review for any correction event scoring below a composite threshold of 3.0. The output schema is designed to be machine-readable for downstream aggregation dashboards; do not remove fields without updating your metrics pipeline accordingly.

This prompt is designed to operate on a single correction event within a multi-turn conversation, not the entire transcript at once. The implementation harness must first identify candidate correction events—turns where the user explicitly contradicts, refines, or rejects a prior assistant statement—and then extract a focused window of turns before and after the correction point. The window should include the assistant turn that contained the original error, the user turn containing the correction, and the subsequent assistant turns (typically 2–4) where the correction should be reflected. Feeding the entire conversation to this prompt dilutes its focus and produces unreliable scores.

Wire the prompt into a batch evaluation pipeline that processes correction events sequentially. For each event, construct the [CONVERSATION_WINDOW] with turn indices and speaker labels, populate [CORRECTION_DETAILS] with the specific user correction text and the assistant statement it targets, and provide the [EVALUATION_RUBRIC] defining your scoring dimensions (acknowledgment, incorporation, persistence, regression). The output [OUTPUT_SCHEMA] should enforce a JSON structure with correction_response_score (numeric), acknowledgment_flag (boolean), incorporation_evidence (string with turn references), regression_detected (boolean), and failure_mode (enum: none, ignored, partial, reverted, contradicted). Validate this schema immediately after model response using a JSON schema validator—malformed outputs should trigger a single retry with the validation error included in the retry prompt.

For production deployment, log every evaluation result with the conversation ID, correction turn index, model version, and prompt version. Store raw scores for trend analysis: if correction-response scores degrade after a model or prompt update, you need per-correction granularity to isolate the regression. Set a score threshold (e.g., below 0.7 on a 0–1 scale) that triggers human review of the correction event. Do not auto-escalate based on a single low score—aggregate across multiple correction events per session before flagging a conversation for review. For high-stakes domains like healthcare or finance, require human review on every correction event where regression_detected is true, regardless of the numeric score.

Model choice matters: this evaluation task requires strong instruction-following and nuanced reading of conversational subtext. Use a capable judge model (GPT-4, Claude 3.5 Sonnet, or equivalent) rather than a smaller, faster model. Latency is acceptable here because correction evaluation is an offline batch process, not a real-time guardrail. If cost is a concern, sample correction events rather than evaluating every one—prioritize sessions with high correction frequency or sessions where the user escalated to a human agent. Avoid using the same model family for both the assistant being evaluated and the judge, as self-evaluation bias can inflate scores.

The most common production failure mode is window misalignment: the harness extracts the wrong turns or includes too much irrelevant context, causing the judge to evaluate the wrong assistant behavior. Build a pre-flight check that verifies the correction turn text actually contains a contradiction or refinement signal before invoking the evaluation prompt. A lightweight classifier prompt (e.g., 'Does this user turn contain a correction to a prior assistant statement?') can gate the evaluation and prevent wasted inference on non-correction turns.