Production Session Replay for Regression Prompt

This prompt is designed for AI ops engineers and platform teams who need to verify that a new system prompt or instruction set does not silently break multi-turn production conversations. The core job-to-be-done is replaying a complete, real-world session trace—including user messages, assistant responses, tool calls, and injected context—through an updated prompt version and producing a structured, turn-by-turn comparison report. It is not a single-turn regression check, a golden dataset replay tool, or a general-purpose output diff. Its value is in detecting cumulative degradation, state drift, and context loss that only emerge over multiple turns in a live production setting.

Use this prompt when you have a complete session trace from a production system. The required inputs are a full conversation history with [USER_MESSAGES], [ASSISTANT_RESPONSES], [TOOL_CALLS], and any [INJECTED_CONTEXT] from the original run. The prompt expects you to provide the new system prompt as [NEW_SYSTEM_PROMPT] and the original as [ORIGINAL_SYSTEM_PROMPT]. The output is a structured regression report with per-turn diffs, state management failure flags, and an aggregate degradation score. This is ideal for teams running canary deployments, pre-release prompt QA, or post-deployment incident reviews where a conversation-level regression is suspected.

Do not use this prompt for single-turn output comparison, initial prompt prototyping, or evaluating a model's general capabilities. It is also not a replacement for a golden dataset test suite; it analyzes a single production trace at a time. Avoid using it on traces that are heavily redacted, missing tool call data, or where the original context injection pipeline has changed, as these gaps will produce false-positive regressions. The next step after reading this section is to prepare your trace data and review the prompt template to understand the exact input schema required.

This template is the core instruction set for an AI ops engineer or an automated harness to replay a captured production session against a new prompt version. It forces a turn-by-turn comparison, tracks state management failures, and computes a cumulative degradation score. Before pasting this into your regression harness, ensure you have access to the original session trace, the new system prompt under test, and any tool definitions that were active during the original session.

textCopy
You are a production regression analyst. Your task is to replay a multi-turn user session against a new prompt version and produce a structured comparison report.

## INPUTS
- Original Session Trace: [ORIGINAL_SESSION_TRACE]
- New System Prompt: [NEW_SYSTEM_PROMPT]
- Tool Definitions: [TOOL_DEFINITIONS]
- User Identity and Session Context: [USER_CONTEXT]

## PROCEDURE
For each turn in the original session trace, do the following:
1. Replay the turn using the new system prompt, maintaining the conversation state from previous replayed turns.
2. Compare the new output against the original output for that turn.
3. Classify the comparison into one of: IDENTICAL, SEMANTICALLY_EQUIVALENT, MINOR_DIFFERENCE, REGRESSION, or CRITICAL_REGRESSION.
4. If a tool call was made, compare the tool name, arguments, and call order against the original.
5. Detect any state drift by checking if the new model has lost context, contradicted earlier turns, or repeated resolved information.

## OUTPUT_SCHEMA
Return a single JSON object with the following structure:
{
  "session_id": "string",
  "new_prompt_version": "string",
  "overall_regression_score": number, // 0.0 (no regression) to 1.0 (complete failure)
  "cumulative_degradation_detected": boolean,
  "turn_comparisons": [
    {
      "turn_index": number,
      "user_input": "string",
      "original_output": "string",
      "new_output": "string",
      "comparison_classification": "IDENTICAL | SEMANTICALLY_EQUIVALENT | MINOR_DIFFERENCE | REGRESSION | CRITICAL_REGRESSION",
      "difference_summary": "string",
      "tool_call_comparison": {
        "original_tool_calls": [...],
        "new_tool_calls": [...],
        "tool_regression_detected": boolean,
        "tool_regression_details": "string"
      },
      "state_drift_detected": boolean,
      "state_drift_details": "string"
    }
  ],
  "state_drift_summary": "string",
  "regression_inventory": [
    {
      "turn_index": number,
      "severity": "MINOR | MODERATE | CRITICAL",
      "description": "string",
      "affected_component": "string"
    }
  ]
}

## CONSTRAINTS
- Do not fabricate original outputs. Use only the provided [ORIGINAL_SESSION_TRACE].
- Maintain strict turn ordering. Do not skip turns.
- If the new prompt causes a refusal where the original complied, classify as CRITICAL_REGRESSION.
- If the new output contains unsupported claims not present in the original, flag as a regression.
- Report state drift only when context from a prior turn is clearly lost or contradicted.

To adapt this template, replace the square-bracket placeholders with actual data from your tracing system. [ORIGINAL_SESSION_TRACE] should be a serialized JSON array of turn objects, each containing user_input, assistant_output, and any tool_calls. [NEW_SYSTEM_PROMPT] is the complete system prompt you are evaluating. [TOOL_DEFINITIONS] should match the function-calling schema used in production. [USER_CONTEXT] can include user properties, session metadata, or any injected context that was present during the original interaction. If your harness already manages state replay, you can remove the procedural instructions and keep only the comparison and output schema sections.

Before integrating this prompt into an automated pipeline, run it against a small set of known-good and known-regressed sessions to calibrate the overall_regression_score threshold. A score above 0.3 often warrants a manual review; above 0.6 should block a release. Pair this prompt with a JSON schema validator in your harness to catch malformed outputs before they corrupt your regression report. For high-stakes deployments, route any session flagged with a CRITICAL_REGRESSION to a human reviewer before accepting the automated analysis.

The session replay prompt is not a one-off analysis tool—it's a regression detection component that must run reliably in automated pipelines. Wire it into a harness that feeds production session pairs (old prompt trace vs. new prompt trace) through the prompt, validates the output structure, and routes findings to your release gate system. The harness should treat each session replay as a test case: input is a complete multi-turn session with metadata, output is a structured comparison report with turn-level diffs, state drift flags, and a cumulative degradation score.

Build the harness with these concrete components: Input assembly pulls session pairs from your trace store, ensuring matching session IDs and identical user inputs across versions. Prompt execution calls the replay prompt with the session pair and a strict [OUTPUT_SCHEMA] requiring turn-by-turn comparison objects, state drift booleans, and a numeric degradation score. Output validation checks schema compliance, verifies turn counts match, confirms degradation scores fall within expected ranges, and flags missing required fields. Retry logic handles transient API failures with exponential backoff (max 3 attempts) but does not retry on validation failures—those indicate prompt or input problems that need human investigation. Result routing pushes validated reports to your regression dashboard, tags sessions exceeding degradation thresholds for review, and blocks release promotion if aggregate degradation crosses your defined gate threshold.

Critical failure modes to instrument: sessions where the new prompt produces identical outputs but the degradation score is non-zero (false positive drift), sessions with legitimate regressions that the prompt misses (false negatives—catch these by spot-checking a random sample against human review), and sessions where state drift accumulates silently across turns without triggering turn-level flags. Log every session replay with the prompt version pair, degradation score, and validation status. Before relying on this harness for release decisions, run it against a curated set of known-regression and known-clean session pairs to calibrate your degradation thresholds and validate that the prompt's scoring aligns with human judgment of regression severity.