User Correction Event Analysis Prompt

Use this prompt when a user explicitly corrects a model's output in a conversational AI product and you need to diagnose the root cause. The ideal user is a conversational AI developer, support engineer, or AI SRE who has already captured a correction event and isolated the relevant trace spans. The job-to-be-done is not live inference or automated self-correction; it is a post-hoc diagnostic workflow that produces a structured analysis of what went wrong, backed by trace evidence. This prompt is the first step in closing the loop between a user-reported correction and a concrete fix recommendation.

This prompt requires a specific set of inputs to be effective. You must provide the original model response that the user corrected, the user's correction text, and the surrounding trace context, which includes prior turns, retrieved documents, tool calls, and system instructions active at the time of the error. The prompt assumes you have already extracted the correction turn from a larger session trace and formatted it into a single trace event payload. Do not use this prompt on raw, unprocessed trace logs or on correction events where the trace context is incomplete or missing. The output is a structured correction analysis containing the original response, the user correction, trace evidence of the error, a fault attribution (prompt defect, retrieval gap, tool failure, or model error), and a fix recommendation. This output requires human review before any fix is implemented, especially for high-severity faults or production prompt changes.

Avoid using this prompt in scenarios where the correction is ambiguous, the user's intent is unclear, or the trace data is too sparse to support a confident attribution. It is not designed for real-time correction handling during live conversations, for aggregating patterns across multiple correction events, or for automatically applying fixes without review. If you need to triage a user-reported issue before deep diagnosis, use the User-Reported Issue Triage Prompt instead. If you need to correlate feedback signals across many sessions, use the Feedback Signal Correlation Prompt. This prompt is the precise diagnostic instrument you reach for when you have one clear correction event and need to understand exactly what failed.

This prompt template is designed to be pasted directly into your trace analysis tool, LLM playground, or API call. It instructs the model to act as a diagnostic engineer, analyzing a specific trace event where a user corrected the assistant's output. The goal is to produce a structured analysis that identifies why the error occurred, not just what the error was. Before using this prompt, ensure you have extracted the relevant trace spans—including the original model response, the user's correction, the system prompt, any retrieved context, and tool-call logs—from your observability platform. The prompt uses square-bracket placeholders that you must replace with real trace data before execution.

textCopy
You are a diagnostic engineer analyzing a production AI trace. Your task is to investigate a specific event where a user corrected the model's output. Use the provided trace data to determine the root cause of the error and recommend a concrete fix.

## INPUT DATA

### Trace Event Summary
- **Event ID:** [TRACE_EVENT_ID]
- **Timestamp:** [TIMESTAMP]
- **Session ID:** [SESSION_ID]

### System Prompt (at time of event)

[SYSTEM_PROMPT]

codeCopy

### Conversation History (last 3 turns before the correction)

[CONVERSATION_HISTORY]

codeCopy

### Original Model Response (the one the user corrected)

[ORIGINAL_MODEL_RESPONSE]

codeCopy

### User Correction

[USER_CORRECTION]

codeCopy

### Retrieved Context (if RAG was used)

[RETRIEVED_CONTEXT]

codeCopy

### Tool-Call Log (if tools were used)

[TOOL_CALL_LOG]

codeCopy

## ANALYSIS INSTRUCTIONS

1.  **Categorize the Error:** Classify the failure into one of these categories:
    - `prompt_defect`: The system prompt was ambiguous, contradictory, or missing a critical instruction.
    - `retrieval_gap`: Required information was missing from the retrieved context, or irrelevant context was included.
    - `tool_failure`: A tool was called incorrectly, returned an error, or the wrong tool was selected.
    - `model_error`: The model hallucinated, ignored a clear instruction, or reasoned incorrectly despite correct inputs.
    - `state_management_error`: The model lost track of the conversation history or user intent.

2.  **Identify the Root Cause:** Pinpoint the exact moment in the trace where the failure originated. Quote the specific line from the system prompt, retrieved context, tool log, or model response that is the primary evidence.

3.  **Assess Severity:** Rate the severity as `low`, `medium`, `high`, or `critical` based on the user impact.

4.  **Recommend a Fix:** Propose a specific, actionable change. This could be a prompt rewrite, a retrieval pipeline adjustment, a tool schema change, or a model upgrade recommendation. Do not implement the fix; only recommend it.

## OUTPUT FORMAT

Return your analysis as a single JSON object with the following schema. Do not include any text outside the JSON object.

{
  "event_id": "[TRACE_EVENT_ID]",
  "error_category": "prompt_defect | retrieval_gap | tool_failure | model_error | state_management_error",
  "root_cause_evidence": "A direct quote from the input data that proves the root cause.",
  "severity": "low | medium | high | critical",
  "fix_recommendation": "A specific, actionable recommendation for the engineering team.",
  "requires_human_review": true
}

## CONSTRAINTS
- Base your analysis strictly on the provided trace data. Do not speculate beyond the evidence.
- If the evidence is insufficient to determine a single root cause, state the most likely cause and set `requires_human_review` to `true`.
- The `fix_recommendation` must be a concrete change, not a vague suggestion like "improve the prompt."

After pasting the template, your primary task is to replace the square-bracket placeholders with actual data from your tracing system (e.g., LangSmith, Arize, Weights & Biases, or a custom log). The quality of the analysis depends entirely on the completeness of this input data. A common failure mode is providing an incomplete trace—for example, omitting the RETRIEVED_CONTEXT in a RAG system—which will force the model to guess and produce an unreliable analysis. Always set requires_human_review to true in the output schema; this prompt is a diagnostic aid, not an auto-remediation tool. The recommended fix must be validated by a human engineer before it is deployed to production.

This prompt is designed to be called programmatically, not used in a one-off chat interface. The primary integration point is a trace-analysis pipeline or an internal support tool that surfaces user-reported corrections. When a user flags an incorrect response, the application must fetch the full trace for that session, extract the relevant spans (user message, model response, tool calls, retrieved context), and inject them into the prompt's placeholders. The prompt is stateless: every call must receive the complete trace evidence it needs. Do not rely on the model remembering prior turns or session state.

Wire the prompt into a structured workflow with clear pre- and post-processing steps. Before calling the model, validate that the trace payload is complete: confirm that the [ORIGINAL_MODEL_RESPONSE] and [USER_CORRECTION] fields are non-empty, and that [TRACE_EVIDENCE] contains at least the generation span and any retrieval or tool-call spans that preceded it. If the trace is incomplete, abort and surface a data-gap error to the operator rather than asking the model to guess. After receiving the model's output, parse the JSON response and validate the schema: correction_category, root_cause, trace_evidence_for_error, and fix_recommendation must all be present and non-null. If validation fails, retry once with a stricter schema reminder appended to the prompt. Log every analysis result—including the trace ID, the model's output, and the validator result—to an audit table for later review.

This workflow is high-risk because the fix recommendation could be applied to a production prompt or retrieval config. Always route the output to a human review queue before any change is implemented. The fix_recommendation field is a suggestion, not an auto-apply instruction. Pair this prompt with an eval step: periodically run a set of known correction events with ground-truth root causes through the pipeline and measure whether the model's correction_category and root_cause match the expected labels. Track precision and recall per category (prompt defect, retrieval gap, tool failure, model error) to detect drift. Choose a model with strong instruction-following and JSON output discipline—GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are reasonable defaults. Avoid smaller or older models that may conflate categories or hallucinate trace evidence not present in the input.

For RAG-heavy applications, ensure the [TRACE_EVIDENCE] includes the retrieved chunks, their ranking scores, and any truncation markers. The model cannot diagnose a retrieval gap if it cannot see what was retrieved. If your trace system captures only the final context window, augment the input with a pre-retrieval summary showing what was available in the index. For tool-use failures, include the tool schema, the arguments passed, and any error codes or timeouts. The prompt's diagnostic accuracy depends entirely on the completeness of the trace evidence you provide. When integrating with a support ticket system, attach the analysis output as an internal note to the ticket and link back to the raw trace for auditors. Do not expose the raw analysis to end users without review—the model may produce plausible-sounding but incorrect attributions.

Finally, treat this prompt as a product component with its own versioning and change log. When you update the prompt template, run your eval suite against the new version before deployment. Monitor the human-review acceptance rate: if reviewers are frequently overriding the model's correction_category or root_cause, the prompt may need refinement or the trace evidence may be insufficient. Set an alert if the validation failure rate exceeds 5% of calls, as this indicates either a schema drift in the model's output or a systemic trace-data gap. The goal is a reliable diagnostic tool that reduces mean time to root cause for user-reported corrections, not a fully automated fix pipeline.