Feedback Signal Correlation Prompt for Model Behavior

This prompt is designed for AI product teams, quality engineers, and observability engineers who need to move beyond treating each user report as an isolated incident. The core job-to-be-done is connecting explicit and implicit user feedback signals—such as thumbs down, manual corrections, session abandonment, or direct complaints—to specific model behaviors recorded in production traces. The output is a correlation matrix that links feedback types to trace patterns, enabling teams to identify systemic failures like a recurring retrieval gap or a brittle prompt instruction that fails under specific conditions. This is a batch analysis workflow, not a real-time diagnostic tool.

Use this prompt when you have a collection of user sessions with associated feedback signals and corresponding production traces that capture the full request context: model inputs, outputs, tool calls, retrieval steps, and state transitions. You must also have a labeled ground-truth dataset to validate the correlation output. Without ground-truth labels, the prompt's statistical confidence notes become unreliable, and you risk confirming spurious correlations. The prompt expects structured input that maps each session ID to its feedback type, trace data, and any known root-cause labels for evaluation. If your traces are incomplete or missing key spans like retrieval calls or tool arguments, the correlation analysis will have blind spots.

Do not use this prompt for real-time alerting or single-session diagnosis. For live debugging of a specific user complaint, use the User Feedback to Trace Mapping Prompt to isolate the exact trace events. For deep-dive analysis of one negative session, use the Negative Feedback Trace Deep-Dive Prompt. This prompt is optimized for aggregate pattern detection across multiple sessions. Also avoid using it when your feedback signals are too sparse to form meaningful clusters—correlation analysis requires a minimum volume of similar feedback events to produce statistically useful output. If you are unsure whether the pattern is real or noise, run the Feedback Cluster to Trace Pattern Matching Prompt first to validate that your feedback clusters have enough density.

This prompt template is designed to be copied directly into your AI harness and adapted with your specific feedback taxonomy, trace event categories, and session datasets. It instructs the model to act as an observability analyst, systematically linking user feedback signals to trace patterns and producing a structured JSON output that can be ingested by downstream monitoring dashboards, alerting systems, or automated remediation pipelines. The template includes placeholders for all required inputs, statistical thresholds, and output constraints, making it suitable for both ad-hoc investigation and automated batch processing of production traces.

textCopy
You are an observability analyst specializing in AI system diagnostics. Your task is to correlate user feedback signals with model behavior captured in production traces. You will receive a dataset of user sessions, each containing a feedback signal and a corresponding trace. You must produce a correlation matrix that links feedback types to trace patterns, with statistical confidence notes.

## INPUT DATA

### Feedback Taxonomy
[FEEDBACK_TAXONOMY]

### Trace Event Categories
[TRACE_EVENT_CATEGORIES]

### Session Dataset
[SESSIONS_DATASET]

### Ground-Truth Labeled Dataset (for validation only, do not train on this)
[GROUND_TRUTH_DATASET]

## INSTRUCTIONS

1. For each session in the dataset, extract the feedback signal and classify it according to the provided feedback taxonomy.
2. For each session, identify the trace events that occurred within the session window and classify them according to the provided trace event categories.
3. Build a correlation matrix where rows are feedback types and columns are trace event categories. Each cell contains the count of sessions where that feedback type co-occurred with that trace event category.
4. For each cell in the matrix, compute a statistical confidence score using the following method: calculate the pointwise mutual information (PMI) between the feedback type and the trace event category, normalized to a 0-1 scale where 1 indicates strong positive correlation and 0 indicates no correlation above chance.
5. Flag any cell where the correlation is statistically significant (PMI > [SIGNIFICANCE_THRESHOLD]) but the session count is below [MINIMUM_SESSION_COUNT]. These are low-confidence correlations that require more data.
6. Compare your correlation matrix against the ground-truth labeled dataset. Report precision, recall, and F1 score for the top [TOP_N] correlations.
7. Identify any feedback types that do not correlate with any trace event category above the significance threshold. These may indicate feedback signals that are not captured by the current trace instrumentation.

## OUTPUT FORMAT

Return a valid JSON object conforming to the output contract specified below. Do not include any text outside the JSON object.

## CONSTRAINTS

- Do not invent feedback signals or trace events not present in the input data.
- Do not report correlations with fewer than [MINIMUM_SESSION_COUNT] sessions as actionable.
- If the ground-truth dataset contains fewer than [MINIMUM_GROUND_TRUTH_SAMPLES] labeled examples, include a warning in the output and reduce confidence scores accordingly.
- All confidence scores must be accompanied by the computation method used.

To adapt this template for your production environment, replace each square-bracket placeholder with your actual data and configuration values. The [FEEDBACK_TAXONOMY] should be a structured list of feedback types you track, such as thumbs_down, correction, abandonment, or escalation. The [TRACE_EVENT_CATEGORIES] should map to the event types your instrumentation captures, like tool_call_failure, retrieval_empty, refusal_triggered, or latency_spike. The [SESSIONS_DATASET] must be a JSON array of session objects, each containing a session_id, feedback_signal, and trace_events array. Set [SIGNIFICANCE_THRESHOLD] to a value like 0.3 for initial exploration and tighten to 0.5 or higher for production alerting. Set [MINIMUM_SESSION_COUNT] to at least 10 to avoid acting on noise, and [MINIMUM_GROUND_TRUTH_SAMPLES] to 50 for reliable validation. Before deploying this prompt into an automated pipeline, validate the output JSON against a schema that enforces the expected structure, and log any parse failures for human review. For high-stakes decisions—such as triggering a prompt rollback or blocking a model version—require human approval on the correlation report before action is taken.

This prompt is designed to run as a scheduled batch job, not a real-time API call. Wire it into a workflow that pulls session data from your observability store, runs the correlation, and writes the output to a dashboard or alerting system. The batch design is intentional: feedback signals accumulate over hours or days, and correlation requires enough session volume to produce statistically meaningful results. Running this prompt on every new feedback event would produce noisy, low-confidence correlations that waste engineering time.

Pre-processing requirements are critical before the prompt sees any data. First, normalize all feedback signals to your taxonomy—free-text feedback must be classified by a separate triage prompt before correlation. Second, flatten and categorize trace events so nested spans are reduced to their parent event category. Third, deduplicate sessions: a single user session with multiple feedback signals should be split into separate rows or the dominant signal selected. These steps prevent the model from correlating against inconsistent or duplicate data, which is the most common cause of false-positive correlations in production.

Post-processing starts with parsing the JSON output and validating it against the output contract schema. If the model returns a low-confidence warning due to insufficient ground-truth data, route the results for human review before publishing to any dashboard. Store the correlation matrix with a timestamp and the prompt version used—this enables trend analysis over time and makes it possible to detect when a prompt version change alters correlation behavior. Without versioned storage, you cannot distinguish a real behavioral shift from a measurement artifact.

Retry logic must handle two distinct failure modes. If the model returns malformed JSON, retry once with a stricter output format instruction that emphasizes valid JSON and the exact schema. If the second attempt fails, log the raw output and escalate to the on-call channel—do not silently drop the batch. If the model reports correlations that violate the minimum session count constraint, discard those correlations and flag the session count validation logic for review. This usually indicates a pre-processing bug where sessions were not properly deduplicated or counted.

Human review gates are mandatory for two scenarios. Any correlation flagged as statistically significant but below the minimum session count must be reviewed by a human before being treated as a signal—small sample sizes produce spurious correlations that look compelling but vanish with more data. If the ground-truth validation F1 score is below 0.7, the entire correlation matrix should be reviewed before product decisions are made. A low F1 score means the prompt is finding patterns that do not match your labeled ground truth, and acting on those patterns will lead to wasted engineering effort on false root causes.

Model choice matters for this workflow. Use a model with strong JSON output capabilities and a context window large enough to hold the full session-feedback dataset. If your dataset exceeds the context window, batch the sessions into groups of 50-100 and run the prompt on each group, then merge the correlation matrices with a weighted average. Do not truncate sessions to fit the window—truncation introduces sampling bias that invalidates the statistical confidence notes. For high-volume production systems, consider running this prompt daily on the previous day's sessions and weekly on a larger rolling window to catch slow-emerging patterns.