User Feedback Correlation with Eval Failures Prompt

Product managers and AI reliability engineers use this prompt when they need to connect explicit user-reported dissatisfaction—such as thumbs-down ratings, support tickets, or negative survey responses—to automated evaluation failures captured in production traces. The core job-to-be-done is deciding which eval rubrics deserve tighter thresholds and which are generating false-positive noise that doesn't reflect actual user experience. Without this correlation analysis, teams often waste engineering cycles tuning evals that users don't care about while missing failures that drive churn. The prompt requires a batch of sessions containing both user feedback signals and automated eval scores, and it produces a structured correlation matrix showing alignment strength, invisible failures, and threshold-tuning recommendations.

This prompt is designed for production triage workflows, not real-time decisioning. You should run it when you have accumulated enough sessions to make the correlation statistically meaningful—typically after a feature release, a model migration, or a prompt version change that has generated user feedback. The prompt expects structured input: each session record must include a session identifier, the user feedback signal (type and value), and the full set of eval dimension scores with their pass/fail status. The output schema includes a correlation strength table, a list of eval dimensions with high user-alignment that warrant tighter thresholds, a list of eval dimensions with low user-alignment that may need threshold relaxation or rubric redesign, and a set of sessions where user feedback was negative but all evals passed—these are the most dangerous blind spots. The prompt also flags sessions where evals failed but users gave positive feedback, helping you identify evals that are too strict or measuring dimensions users don't notice.

Do not use this prompt when you lack explicit user feedback signals—implicit signals like session duration or abandonment rates require different analysis and are outside this prompt's scope. Do not use it for real-time per-request routing decisions; the correlation analysis is a batch diagnostic tool meant to inform eval threshold configuration, not a live decision engine. The prompt requires human review of its recommendations before any threshold changes are applied to production evals, especially when the analysis suggests relaxing safety or compliance-related rubrics. After running this prompt, the next step is to take the high-alignment eval dimensions and tighten their pass/fail thresholds in your evaluation pipeline, then monitor whether user dissatisfaction signals decrease in subsequent sessions. For the blind-spot sessions where users were unhappy but evals passed, you should run a separate root-cause investigation prompt to determine whether you need new eval dimensions entirely.

This prompt template is designed to be pasted directly into your LLM interface or API call. It takes structured user feedback records and corresponding automated eval results for the same production traces, then produces a correlation analysis. The analysis identifies which eval dimensions align with user dissatisfaction signals, which eval failures are invisible to users, and where eval thresholds may need tuning. All placeholders are enclosed in square brackets and must be replaced with your actual data before execution.

textCopy
You are an AI observability analyst reviewing production traces where both user feedback and automated evaluation results are available. Your task is to produce a correlation analysis that connects explicit user-reported issues to specific eval dimension failures.

## INPUT DATA

### User Feedback Records
[USER_FEEDBACK_RECORDS]

### Automated Eval Results
[EVAL_RESULTS]

### Eval Rubric Reference
[EVAL_RUBRIC]

## OUTPUT SCHEMA

Produce a JSON object with the following structure:

{
  "correlation_summary": {
    "total_traces_analyzed": <integer>,
    "traces_with_user_feedback": <integer>,
    "traces_with_eval_failures": <integer>,
    "traces_with_both": <integer>,
    "overall_alignment_score": <float between 0 and 1>
  },
  "dimension_correlations": [
    {
      "eval_dimension": "<dimension name from rubric>",
      "user_signal_alignment": "high" | "medium" | "low" | "none",
      "co_occurrence_rate": <float between 0 and 1>,
      "false_positive_rate": <float between 0 and 1>,
      "false_negative_rate": <float between 0 and 1>,
      "representative_examples": [
        {
          "trace_id": "<identifier>",
          "user_feedback_summary": "<brief description>",
          "eval_failure_detail": "<which eval dimension failed and why>",
          "alignment_assessment": "aligned" | "misaligned" | "partially_aligned"
        }
      ],
      "threshold_recommendation": "<specific suggestion or 'no change needed'>"
    }
  ],
  "invisible_failures": [
    {
      "eval_dimension": "<dimension name>",
      "failure_count": <integer>,
      "user_complaint_count": 0,
      "severity_assessment": "low_impact" | "monitor" | "investigate",
      "explanation": "<why users did not report this failure>"
    }
  ],
  "user_reported_issues_without_eval_coverage": [
    {
      "issue_description": "<summary of user complaint>",
      "frequency": <integer>,
      "suggested_new_eval_dimension": "<proposed dimension name>",
      "suggested_rubric_criteria": "<how this could be evaluated automatically>"
    }
  ],
  "threshold_tuning_recommendations": [
    {
      "eval_dimension": "<dimension name>",
      "current_threshold": "<current setting if known>",
      "recommended_adjustment": "increase" | "decrease" | "maintain",
      "rationale": "<evidence-based reasoning>"
    }
  ]
}

## CONSTRAINTS

- Only report correlations where you have sufficient data to draw a conclusion. Mark low-sample dimensions as "insufficient data" rather than forcing a correlation.
- Distinguish between causation and coincidence. A co-occurrence does not automatically mean the eval failure caused the user complaint.
- For threshold recommendations, consider the cost of false positives (flagging acceptable outputs) versus false negatives (missing real user pain).
- If user feedback is vague (e.g., "bad response"), do not force a mapping to a specific eval dimension. Note it as "uncorrelated feedback."
- Preserve trace IDs exactly as provided for downstream trace review.
- If the eval rubric is not provided, infer dimension names from the eval results structure and note the inference.

## ANALYSIS STEPS

1. Parse all user feedback records and eval results, matching them by trace ID.
2. For each eval dimension, calculate the co-occurrence rate with user complaints.
3. Identify eval failures that never co-occur with user complaints (invisible failures).
4. Identify user complaints that do not map to any existing eval dimension (coverage gaps).
5. For each dimension, assess whether the current eval threshold is calibrated to user-perceived quality.
6. Produce the output JSON with evidence-backed recommendations.

To adapt this template, replace each square-bracket placeholder with your actual data. [USER_FEEDBACK_RECORDS] should contain structured records including trace IDs, feedback text, sentiment, and any user-provided ratings. [EVAL_RESULTS] should contain per-trace eval scores broken down by dimension. [EVAL_RUBRIC] should describe each eval dimension, its scoring criteria, and current pass/fail thresholds. If your eval system uses different terminology, adjust the dimension names in the output schema accordingly. For high-stakes production systems, run this analysis on a representative sample before applying threshold changes broadly, and always have a human review the threshold tuning recommendations before implementation.

This prompt is designed to operate on structured inputs, not raw logs. Before calling the model, you must assemble a correlation dataset from two production sources: your automated evaluation system and your user feedback channels. The eval system should provide per-session scores across defined dimensions (e.g., accuracy, grounding, tone). The feedback system should provide explicit user-reported issues, satisfaction ratings, or support tickets, each tagged with a session or trace ID. The application layer must join these datasets on a common identifier, such as trace_id or session_id, to produce the [CORRELATION_DATASET] input. This pre-processing step is critical; feeding the model raw, unjoined data will produce unreliable correlations.

The implementation harness should enforce a strict schema for the [CORRELATION_DATASET] input. Each record must include session_id, eval_scores (a map of dimension names to numeric scores), eval_pass (boolean), user_feedback_score (numeric or null), and user_issue_reported (boolean). The application should validate this schema before calling the model and reject any malformed records. For the output, parse the model's JSON response and validate it against the expected [OUTPUT_SCHEMA]. A retry loop with a maximum of 2 attempts is appropriate for JSON parse failures. If the model's output fails schema validation after retries, log the raw output and escalate for manual review rather than silently dropping the analysis.

Model choice matters for this workflow. The prompt requires cross-referencing numerical scores with categorical feedback signals, a task where models with strong reasoning capabilities (e.g., Claude 3.5 Sonnet, GPT-4o) outperform smaller or faster models. Avoid routing this prompt to a lightweight model optimized for latency; the cost of a wrong correlation—such as tuning an eval threshold based on a spurious relationship—is higher than the inference cost. If your observability stack supports model routing, pin this prompt to a reasoning-capable model and bypass any cost-optimization fallback rules. Log the model version and prompt version alongside each correlation result for auditability.

The most common production failure mode is a mismatch between eval dimensions and user feedback categories. If your eval system scores factual_accuracy but users report issues with unhelpful_tone, the model will correctly report zero correlation, which may appear as a 'failed' analysis. Your harness should distinguish between a true zero-correlation finding and a data quality issue. Implement a pre-flight check that counts the number of sessions where both eval failures and user-reported issues exist. If this count is below a configurable threshold (e.g., fewer than 5 correlated failure events), surface a warning in the application layer rather than calling the model, as the sample size is too small for meaningful correlation analysis.

Finally, treat the model's output as an investigative lead, not a final decision. The correlation analysis identifies which eval dimensions align with user dissatisfaction, but it does not prove causation. After receiving the model's output, the next step in your workflow should be a focused trace review of the sessions where eval failures and user reports co-occur. Use the Eval Failure Root-Cause Triage Prompt or Trace Review and Diagnosis Prompts from this pillar to drill into those specific sessions. Never automatically adjust eval thresholds based solely on this correlation output without human review of the underlying traces.