Negative Feedback Trace Deep-Dive Prompt

This prompt is designed for quality engineers and AI SREs who need to perform a structured failure analysis on a single production trace that has been explicitly linked to negative user feedback. The core job-to-be-done is to move from a vague user complaint—such as 'the answer was wrong' or 'the assistant got stuck'—to a precise, evidence-backed root cause. The ideal user has access to a complete trace, including the system prompt, user input, retrieval steps, tool calls, and the final model output, and needs to produce a diagnostic report that can be shared with the engineering team or attached to an incident ticket.

To use this prompt effectively, you must provide the full trace data as the primary input. The trace should include all spans, not just the final generation. The prompt is built to reconstruct the event chain, identify the exact failure point, and assess contributing factors such as ambiguous instructions, missing retrieval context, or a malformed tool call. It is not designed for aggregate trend analysis across thousands of sessions, nor is it a real-time alerting mechanism. Do not use this prompt when the trace data is incomplete, when key spans are missing, or when you are trying to diagnose a systemic issue from a sample of traces—use the 'Feedback Cluster to Trace Pattern Matching' or 'Incident Trace Correlation' prompts for those scenarios.

Before executing this prompt, confirm that the user feedback is tied to a specific trace ID. If you only have a user email or a timestamp, use the 'User Feedback to Trace Mapping Prompt' first to locate the correct trace. The output of this deep-dive prompt is a detailed failure analysis, not a simple classification. It requires careful human review of the model's recommendations before any code or prompt changes are deployed. Treat the analysis as a diagnostic artifact that informs a fix, not as an automated remediation step.

The following prompt template is designed to be copied directly into your AI harness or observability platform. It forces a structured, evidence-based diagnosis of a single production trace, moving beyond surface-level summaries to identify the precise failure point, contributing factors, and severity. Use this template when a user has explicitly reported a bad experience (thumbs-down, correction, complaint ticket) and you have the corresponding trace data available. Do not use this prompt for aggregate trend analysis or proactive anomaly scanning; it is built for deep-dive, single-trace root-cause isolation.

markdownCopy
You are a senior AI reliability engineer performing a root-cause failure analysis on a single production trace that received negative user feedback. Your goal is to produce a detailed, evidence-backed report that isolates the failure point and explains why it occurred.

## Input Data
- **User Feedback:** [USER_FEEDBACK]
- **Full Trace Data (JSON):** [TRACE_DATA]
- **Expected Behavior (if known):** [EXPECTED_BEHAVIOR]

## Analysis Constraints
- Ground every finding in a specific trace span, event, or log line. Do not speculate without evidence.
- If the trace data is incomplete or ambiguous, state that explicitly and flag it as a limitation.
- Classify the failure into exactly one primary category: `prompt_defect`, `retrieval_gap`, `tool_failure`, `model_error`, `routing_error`, or `infrastructure_failure`.
- Do not propose a fix. Focus strictly on diagnosis and evidence.

## Required Output Schema
Return a valid JSON object with the following structure. Do not include any text outside the JSON.
{
  "trace_id": "string",
  "analysis_timestamp": "string (ISO 8601)",
  "user_feedback_summary": "string (concise restatement of the user's complaint)",
  "failure_category": "prompt_defect | retrieval_gap | tool_failure | model_error | routing_error | infrastructure_failure",
  "failure_point": {
    "span_id": "string (the specific trace span where the failure originated)",
    "event_type": "string (e.g., 'llm_call', 'tool_execution', 'retrieval_query')",
    "description": "string (what went wrong at this exact point)"
  },
  "event_chain": [
    {
      "sequence": "integer",
      "span_id": "string",
      "event_summary": "string",
      "was_successful": "boolean",
      "anomaly_flag": "boolean (true if this event deviated from expected behavior)"
    }
  ],
  "contributing_factors": [
    {
      "factor": "string (description of a condition that contributed to the failure)",
      "evidence_span_id": "string (trace span supporting this factor)",
      "impact": "high | medium | low"
    }
  ],
  "severity_assessment": {
    "user_impact": "blocking | degraded | cosmetic",
    "reproducibility_likelihood": "high | medium | low",
    "rationale": "string (justification for the severity ratings based on trace evidence)"
  },
  "limitations": ["string (list any gaps in the trace data or analysis uncertainty)"]
}

## Analysis Steps
1. Parse the user feedback to understand the reported symptom.
2. Walk the trace chronologically, building the event chain.
3. Identify the first span where behavior deviated from the expected path.
4. List all contributing factors (e.g., missing context, ambiguous instructions, tool timeout) with trace evidence.
5. Assess severity based on user impact and how likely the failure is to recur.
6. Populate the output JSON completely. Leave no array empty; use "none_identified" for fields with no applicable value.

To adapt this template, replace the placeholders with your actual data. [USER_FEEDBACK] should contain the raw text of the user's complaint, correction, or feedback signal. [TRACE_DATA] expects the full trace JSON from your observability platform (e.g., LangSmith, Arize, Weights & Biases, or a custom trace store). If you have a documented expected behavior or golden trace for comparison, provide it in [EXPECTED_BEHAVIOR]; otherwise, replace it with the string "not_provided". The output schema is strict JSON, so ensure your application layer validates the response against this schema before accepting it. If the model returns malformed JSON, use a retry recovery prompt that feeds the raw output and schema back to the model for repair.

Before deploying this prompt into an automated pipeline, run it against a labeled dataset of 10-20 traces with known failure categories. Measure whether the failure_category field matches the ground-truth label and whether the failure_point.span_id correctly identifies the originating span. A human reviewer should sign off on the first 50 analyses to calibrate severity assessments and catch hallucinated evidence citations. For high-severity incidents (user_impact: "blocking"), always require human approval before the analysis is shared with stakeholders or used to trigger an automated rollback.

This prompt is designed to be called programmatically as part of an automated trace analysis pipeline, not as a one-off chat interaction. The primary integration point is a webhook or queue consumer that triggers when a user submits negative feedback (e.g., a thumbs-down, a support ticket, or an escalation). The system must first retrieve the full trace payload—including all spans, tool calls, retrieved documents, and model responses—from your observability store (such as LangSmith, Arize, or a custom logging database) and inject it into the [FULL_TRACE_JSON] placeholder. Without the complete trace, the analysis will be speculative and unreliable.

Before calling the LLM, validate the input trace for structural completeness: confirm that required fields like trace_id, spans, start_time, and feedback_signal are present and non-null. If the trace is malformed or truncated, abort the deep-dive and route to a human analyst with a TRACE_INCOMPLETE error code. For the model call itself, use a low-temperature setting (0.0–0.2) to enforce deterministic, evidence-grounded analysis. Implement a retry wrapper with exponential backoff (max 3 attempts) to handle transient API failures, but do not retry on validation errors or content policy refusals—those should be logged and escalated immediately. The output must be parsed as strict JSON and validated against the expected [OUTPUT_SCHEMA] before it is stored or displayed. If the output fails schema validation, invoke a repair prompt or fall back to a structured error log that captures the raw response for manual review.

Log every invocation—including the trace ID, feedback signal, model used, latency, token counts, and the full analysis output—to an audit table. This log serves as evidence for compliance reviews and allows you to measure the accuracy of the automated analysis over time. For high-severity findings (severity critical or high), route the analysis to a human reviewer via your ticketing system before any automated fix is applied. Never allow the prompt's recommended fix to be deployed directly to production without human approval. Finally, build an eval harness that periodically tests the prompt against a golden set of annotated traces with known failure modes to detect drift in analysis quality.