Feedback-to-Prompt-Defect Detection Prompt

This prompt is designed for prompt engineers and AI reliability engineers who need to determine whether a piece of negative or corrective user feedback points to a defect in the system prompt itself. The job-to-be-done is isolating the root cause of a behavioral failure when the model's output was technically valid but violated user expectations, policy, or the intended assistant contract. The ideal user is someone who already has access to a production trace containing the user feedback signal, the full system instructions, and the model's final response. They are not guessing; they are performing a structured differential diagnosis to separate prompt ambiguity from retrieval gaps, tool-call errors, or model hallucinations.

You should use this prompt when user feedback is explicit (e.g., a thumbs-down, a correction like 'you shouldn't have done that,' or a complaint that the assistant ignored a rule) and the root cause is unclear from a surface reading of the trace. The prompt works by forcing the model to compare the system instructions against the assistant's behavior, identify the specific instruction that was violated, was ambiguous, or conflicted with another rule, and produce a defect report with a suggested rewrite. Do not use this prompt for failures where the model hallucinated facts not present in the prompt, where a tool call returned an error, or where the retrieval pipeline returned irrelevant documents. Those failures require their own diagnostic prompts. This prompt is strictly for instruction-level defects.

Before using this prompt, ensure you have the complete system prompt, the user's feedback signal, and the model's response captured in the trace. The output requires human review before any prompt change is deployed. A suggested rewrite from this prompt is a diagnostic artifact, not a deployable fix. You must validate that the rewrite does not introduce new conflicts, regress other behaviors, or violate the system's broader instruction hierarchy. Pair this prompt with a regression test run against your golden dataset before accepting any change.

This template is designed to be pasted directly into your trace analysis tool, evaluation harness, or LLM workbench. It instructs the model to act as a prompt forensics engineer, correlating a specific piece of user feedback with the raw trace data to isolate whether the root cause is a defect in the system prompt's instructions. The prompt forces the model to cite specific trace events as evidence and propose a concrete, minimal prompt rewrite, rather than offering vague suggestions.

textCopy
You are a prompt forensics engineer. Your task is to analyze a user feedback report against a production trace to determine if the root cause is a defect in the system prompt.

<USER_FEEDBACK>
[USER_FEEDBACK]
</USER_FEEDBACK>

<PRODUCTION_TRACE>
[PRODUCTION_TRACE]
</PRODUCTION_TRACE>

<SYSTEM_PROMPT>
[SYSTEM_PROMPT]
</SYSTEM_PROMPT>

Follow these steps:
1.  **Analyze the Feedback:** Paraphrase the user's core complaint and the expected behavior.
2.  **Reconstruct the Failure:** Identify the exact point in the <PRODUCTION_TRACE> where the model's behavior diverged from the expected behavior. Cite the specific trace span ID, tool call, or log line.
3.  **Diagnose the Root Cause:** Determine if the failure is a prompt defect, a retrieval gap, a tool failure, or a model reasoning error. If it is a prompt defect, identify the exact conflicting, ambiguous, or missing instruction in the <SYSTEM_PROMPT>.
4.  **Propose a Prompt Rewrite:** If a prompt defect is found, output a minimal, targeted rewrite of the offending section. The rewrite must be a drop-in replacement that addresses the defect without changing other behavior.
5.  **Assess Confidence:** Provide a confidence score (High/Medium/Low) for your diagnosis and rewrite.

Your output must be a JSON object conforming to this schema:
{
  "feedback_analysis": "string",
  "failure_reconstruction": {
    "divergence_point": "string (trace span ID or event)",
    "trace_evidence": "string (excerpt from the trace)"
  },
  "root_cause_diagnosis": {
    "category": "PROMPT_DEFECT | RETRIEVAL_GAP | TOOL_FAILURE | MODEL_ERROR",
    "offending_prompt_section": "string | null (exact text from system prompt)",
    "defect_explanation": "string | null (explanation of the ambiguity, conflict, or omission)"
  },
  "prompt_rewrite": {
    "original_text": "string | null",
    "rewritten_text": "string | null",
    "rewrite_rationale": "string | null"
  },
  "confidence": "HIGH | MEDIUM | LOW"
}

To adapt this template, replace the bracketed placeholders with your actual data. [USER_FEEDBACK] should contain the raw text of the user's complaint. [PRODUCTION_TRACE] should be the full, structured trace data (e.g., an OpenTelemetry JSON export or a LangSmith trace). [SYSTEM_PROMPT] is the exact version of the system prompt that was active when the trace was captured. The output schema is strict JSON, making it suitable for direct integration into an automated triage pipeline. For high-risk applications, always route the prompt_rewrite object to a human-in-the-loop review queue before deploying the change.

This prompt is designed to operate as a diagnostic step within a larger trace-analysis pipeline, not as a standalone chat interaction. The typical trigger is a user-reported issue or a negative feedback signal (thumbs-down, correction, abandonment) that has already been mapped to a specific production trace via a preceding 'User Feedback to Trace Mapping' step. The harness should fetch the full trace payload—including the system prompt version, user messages, retrieved context chunks, tool-call definitions, and the final model output—and inject it into the [TRACE_JSON] placeholder. The [FEEDBACK_SIGNAL] placeholder should receive a structured description of what the user reported, not raw free-text, to improve consistency.

Validation and retry logic is critical because this prompt produces a structured defect report that downstream systems may act on. After the model returns its output, validate the JSON schema strictly: the offending_prompt_section field must be a non-empty string that can be located in the original system prompt, the trace_evidence array must contain at least one span ID or timestamp from the input trace, and the suggested_rewrite must differ from the original text. If validation fails, retry once with the validation errors appended to the [CONSTRAINTS] field. If the second attempt also fails, escalate to a human reviewer rather than silently accepting a malformed report. For high-severity classifications (where defect_severity is 'critical' or 'high'), always require human approval before the suggested rewrite is deployed to production, regardless of validation success.

Model choice and tool use should favor models with strong instruction-following and structured output capabilities. GPT-4o, Claude 3.5 Sonnet, or equivalent models are appropriate. Enable structured output mode (JSON mode or function calling with a strict schema) to reduce parsing errors. This prompt does not require tool calls or retrieval during execution—the trace data is provided inline—but the harness should log the model's output, the validation result, and the human review decision to an audit table for governance. Avoid wiring this prompt directly to an automated prompt-deployment pipeline; the suggested rewrite is a recommendation, not a command. The final step should always be a human-in-the-loop review that considers the defect report, the suggested rewrite, and any regression test results before merging the change.