Model Change Impact Trace Comparison Prompt

This prompt is for incident responders and AI SREs investigating a suspected model update. When a provider rolls out a new model version or an internal model is swapped, behavioral changes can surface as output structure drift, tool selection errors, refusal rate spikes, or latency regressions. This prompt compares pre-change and post-change traces for the same or similar inputs, isolates behavioral diffs, and produces a side-by-side change report with confidence scores. It requires grounding in specific trace pairs to avoid spurious correlation.

Use this prompt when you have access to production trace data from before and after a known or suspected model change and need to determine whether the model update caused the incident you are investigating. The prompt expects paired traces—ideally the same or semantically equivalent inputs processed by the old and new model versions. It works best when traces include full spans: system prompt, user input, tool calls and their arguments, model output, and latency measurements. Without paired traces, the comparison degrades to a heuristic analysis with lower confidence, and the prompt will flag this limitation in its output.

Do not use this prompt when you lack pre-change traces, when the incident is clearly an infrastructure failure (timeouts, network errors, OOM), or when you need real-time alerting rather than post-hoc investigation. This prompt is an investigative tool, not a monitoring query. For continuous monitoring, pair it with a trace-based alert that triggers when behavioral fingerprints shift beyond a threshold. Always review the prompt's confidence scores before escalating findings to a stakeholder-facing incident report. Low-confidence diffs should be treated as hypotheses requiring further trace collection, not as confirmed root causes.

This template is the core diagnostic instrument for an incident responder investigating a suspected model change. It is designed to be pasted directly into your incident response workflow, where it will consume two structured trace payloads—one from before the change and one from after—for the same or semantically equivalent input. The prompt instructs the model to perform a structured, side-by-side comparison, isolating diffs in output structure, tool selection, refusal rate, and latency, and to produce a change report with confidence scores. The output is intended to be machine-readable for downstream automated rollback or escalation decisions.

markdownCopy
You are an AI incident responder specializing in model behavior forensics. Your task is to compare two production traces—one from before a suspected model update and one from after—for the same or a semantically equivalent user input. Your goal is to isolate behavioral changes attributable to the model update, not to input variance or environmental noise.

## INPUT DATA
- Pre-Change Trace: [PRE_CHANGE_TRACE]
- Post-Change Trace: [POST_CHANGE_TRACE]
- User Input: [USER_INPUT]
- Evaluation Context: [EVALUATION_CONTEXT]

## COMPARISON DIMENSIONS
For each dimension below, provide a structured finding with a `change_detected` boolean, a `confidence` score (0.0 to 1.0), and a `diff_summary` string.

1. **Output Structure**: Compare the top-level structure of the final output (e.g., JSON keys, markdown sections, list formats). Note any added, removed, or reordered elements.
2. **Tool Selection**: Compare the sequence of tool calls, including tool names, argument schemas, and call order. Flag any new, missing, or reordered tool invocations.
3. **Refusal Behavior**: Compare whether the model refused to answer, asked for clarification, or responded with a safety disclaimer. Note any change in refusal style or threshold.
4. **Latency Profile**: Compare the total response time and per-step latency (time-to-first-token, tool-call duration, inter-tool delays). Flag any step with a >20% latency delta.
5. **Content Fidelity**: Compare the factual accuracy and completeness of the final response relative to the provided input and context. Note any new hallucinations or omissions.

## OUTPUT SCHEMA
Return a single JSON object conforming to this exact structure:
{
  "analysis_id": "string",
  "pre_change_trace_id": "string",
  "post_change_trace_id": "string",
  "user_input_hash": "string",
  "overall_change_detected": boolean,
  "overall_confidence": number,
  "dimensions": {
    "output_structure": {
      "change_detected": boolean,
      "confidence": number,
      "diff_summary": "string"
    },
    "tool_selection": {
      "change_detected": boolean,
      "confidence": number,
      "diff_summary": "string"
    },
    "refusal_behavior": {
      "change_detected": boolean,
      "confidence": number,
      "diff_summary": "string"
    },
    "latency_profile": {
      "change_detected": boolean,
      "confidence": number,
      "diff_summary": "string"
    },
    "content_fidelity": {
      "change_detected": boolean,
      "confidence": number,
      "diff_summary": "string"
    }
  },
  "spurious_correlation_risks": ["string"],
  "recommendation": "ROLLBACK | INVESTIGATE_FURTHER | NO_ACTION"
}

## CONSTRAINTS
- Ground every finding in specific spans or log lines from the provided traces. Do not speculate beyond the evidence.
- If the two traces have different user inputs, flag this as a high spurious correlation risk and reduce confidence scores accordingly.
- If any dimension cannot be evaluated due to missing data, set `change_detected` to false, `confidence` to 0.0, and explain the data gap in `diff_summary`.
- Do not invent benchmark numbers, model version strings, or customer impact statements.

To adapt this template for your environment, replace the [PRE_CHANGE_TRACE] and [POST_CHANGE_TRACE] placeholders with serialized trace objects from your observability store. The [USER_INPUT] field should contain the exact user message or API payload sent to both traces. The [EVALUATION_CONTEXT] field is optional but recommended for high-risk domains; use it to inject relevant policy rules, expected behavior contracts, or golden-set references. For automated incident pipelines, parse the recommendation field to trigger a rollback if ROLLBACK is returned with overall_confidence above your threshold (we suggest 0.85). Always log the full prompt and response for post-incident review and auditability.

This prompt is designed to be invoked programmatically during an incident investigation, not as a one-off chat interaction. The implementation harness must fetch pre-change and post-change trace pairs from your observability store, format them into the prompt's [PRE_CHANGE_TRACES] and [POST_CHANGE_TRACES] placeholders, and handle the structured output for downstream analysis. Because the prompt compares behavioral diffs across a model update boundary, the harness must ensure that trace pairs are matched on equivalent or semantically similar inputs—mismatched pairs will produce spurious correlation in the change report. The harness should query your trace database with filters for model_version, deployment_timestamp, and input_hash or semantic_cluster_id to construct valid comparison pairs before invoking the model.

The integration follows a fetch → validate → prompt → parse → escalate loop. First, the harness retrieves trace pairs within the incident window and validates that each pair contains complete spans (tool calls, final output, latency metrics, refusal flags). Incomplete traces are discarded or flagged for human review. Second, the harness constructs the prompt with the trace pairs, [COMPARISON_DIMENSIONS] (output structure, tool selection, refusal rate, latency), and [CONFIDENCE_THRESHOLD] (default 0.7). Third, the model returns a structured JSON change report with per-dimension diffs and confidence scores. The harness must parse this output against a strict schema—rejecting malformed JSON and retrying with a repair prompt if validation fails. Fourth, any diff with a confidence score below the threshold or flagged as requires_human_review: true is routed to an incident responder via your alerting channel (PagerDuty, Slack, incident ticket). Log every invocation—trace IDs, model version, prompt hash, output, and review decisions—for post-incident audit and prompt improvement.

For model choice, use a model with strong structured output capabilities and a context window large enough to hold multiple full trace payloads. GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are appropriate; avoid smaller models that may drop spans or hallucinate diff details. Set temperature=0 to maximize reproducibility across incident responders. For retry logic, implement up to two retries on schema validation failure, each time appending the validation error to the prompt as a repair instruction. If the model consistently fails to produce valid output, escalate the raw trace pairs to a human with a note that automated comparison failed. For cost control, consider pre-filtering traces to only those with anomaly scores above a threshold before invoking the prompt, reducing token consumption during wide-scale incidents. Do not use this prompt for real-time gating on model deployments—the latency of trace retrieval and model inference makes it unsuitable for blocking rollouts; instead, use it for post-deployment investigation within the incident response window.

Avoid wiring this prompt directly to an automated rollback action. The confidence scores are diagnostic signals, not deployment controls. A low-confidence diff may indicate noisy data rather than a real regression, and an automated rollback triggered by a false positive can compound an incident. Always require human acknowledgment before acting on the change report. For teams running continuous monitoring, schedule a periodic (e.g., hourly) invocation during the post-deployment observation window, comparing the last N pre-change traces against the first N post-change traces, and route the report to the on-call channel as a proactive diagnostic artifact rather than an alert.