Canary Deployment Output Comparison Prompt Template

This prompt is designed for MLOps engineers and platform teams who are running a canary deployment of a new prompt version. The core job-to-be-done is to compare the outputs of a stable baseline prompt against a new canary prompt using a shared evaluation dataset. The goal is not just to see if the outputs are different, but to produce a structured, machine-readable diff report that quantifies semantic drift, flags regressions, and provides a data-driven recommendation to promote or roll back the canary. The ideal user has a golden dataset of inputs and expected behaviors ready, and they need an automated gate decision to integrate into a CI/CD pipeline, not a manual review of a few samples.

You should use this prompt when you have a controlled canary environment where both the baseline and canary prompts can be run against the same set of inputs. The prompt assumes you can provide the raw outputs from both versions, along with any relevant context like the evaluation rubric or expected output schema. It is particularly useful when the risk of a bad deployment is high—for example, in customer-facing chat, regulated document processing, or any system where a subtle change in tone, refusal rate, or factual grounding could cause user harm or operational incidents. The prompt is built to catch failures that simple string matching or regex checks would miss, such as a model becoming more verbose but less accurate, or shifting its refusal style in a way that breaks a downstream parser.

Do not use this prompt as a replacement for full regression test suites or human review in high-stakes domains. It is a comparative analysis tool, not a root-cause analysis engine. If your canary and baseline outputs are generated from different models or significantly different system prompts, the semantic drift scores may be noisy and require calibration. Avoid using this prompt when you lack a representative evaluation dataset; comparing outputs on a handful of cherry-picked examples will produce a misleading confidence score. After running this prompt, the structured recommendation should be fed into an automated gate with predefined thresholds, and any 'rollback' or 'manual review' decision should be logged as evidence for your deployment audit trail.

This template is designed for MLOps engineers running canary prompt deployments. It expects a set of shared inputs, the corresponding outputs from both the baseline and canary prompt versions, and a defined evaluation rubric. The prompt instructs the model to act as a structured diff engine, focusing on semantic differences, instruction adherence, and safety-critical regressions rather than superficial lexical changes. The output is a machine-readable report that can be consumed by an automated promotion gate or a human reviewer.

textCopy
You are an AI output comparison engine for canary prompt deployments. Your task is to compare outputs from a BASELINE prompt and a CANARY prompt across a shared evaluation dataset. Produce a structured diff report that quantifies semantic drift, identifies regressions, and recommends a deployment action.

## INPUTS
- EVALUATION DATASET: [EVAL_DATASET]
- BASELINE OUTPUTS: [BASELINE_OUTPUTS]
- CANARY OUTPUTS: [CANARY_OUTPUTS]
- EVALUATION RUBRIC: [EVAL_RUBRIC]
- DRIFT THRESHOLDS: [DRIFT_THRESHOLDS]
- REGRESSION SEVERITY MAP: [REGRESSION_SEVERITY_MAP]

## OUTPUT SCHEMA
Return a single JSON object with the following structure:
{
  "comparison_id": "string",
  "baseline_version": "string",
  "canary_version": "string",
  "dataset_size": integer,
  "comparisons": [
    {
      "example_id": "string",
      "input_summary": "string",
      "semantic_drift_score": float (0.0-1.0),
      "drift_category": "none" | "cosmetic" | "minor_rewording" | "semantic_shift" | "instruction_violation" | "safety_regression",
      "key_differences": ["string"],
      "regression_flag": boolean,
      "regression_severity": "low" | "medium" | "high" | "critical" | null,
      "regression_type": "schema_violation" | "hallucination_increase" | "refusal_change" | "tone_degradation" | "accuracy_drop" | "latent_risk" | null,
      "rubric_score_baseline": float,
      "rubric_score_canary": float,
      "recommendation": "accept" | "review" | "reject"
    }
  ],
  "aggregate_metrics": {
    "mean_semantic_drift": float,
    "regression_count": integer,
    "critical_regression_count": integer,
    "rubric_score_delta_mean": float,
    "examples_improved": integer,
    "examples_degraded": integer,
    "examples_unchanged": integer
  },
  "deployment_recommendation": "promote" | "hold_for_review" | "rollback",
  "recommendation_confidence": float (0.0-1.0),
  "risk_summary": "string",
  "requires_human_review": boolean,
  "human_review_examples": ["string"]
}

## CONSTRAINTS
- Compare outputs semantically, not lexically. Two outputs with different wording but identical meaning should have a low drift score.
- Flag any example where the canary output violates the evaluation rubric, even if the baseline also violated it.
- A regression is defined as a canary output that is measurably worse than the baseline on rubric criteria.
- If any critical regression is detected, the deployment recommendation must be "rollback" or "hold_for_review".
- Do not hallucinate differences. If two outputs are functionally identical, set drift_score to 0.0 and drift_category to "none".
- For each comparison, provide at least one specific, quoted difference if drift_category is not "none".
- The risk_summary must be a concise, human-readable paragraph explaining the primary risks and the reasoning behind the deployment recommendation.

## EVALUATION STEPS
1. For each example in the dataset, read the input, baseline output, and canary output.
2. Score both outputs against the evaluation rubric.
3. Identify semantic differences between the two outputs.
4. Classify the drift category based on the nature and severity of the difference.
5. Determine if the difference constitutes a regression.
6. Assign a per-example recommendation.
7. Compute aggregate metrics across all examples.
8. Apply the drift thresholds and regression severity map to produce the final deployment recommendation.
9. Populate the output JSON completely and accurately.

To adapt this template, replace each square-bracket placeholder with your actual data. The [EVAL_DATASET] should contain input examples and optional ground-truth references. The [EVAL_RUBRIC] defines the scoring criteria used to judge both baseline and canary outputs—this is critical for consistent regression detection. The [DRIFT_THRESHOLDS] and [REGRESSION_SEVERITY_MAP] allow you to encode your organization's risk tolerance directly into the prompt. Before wiring this into an automated pipeline, validate the output JSON against the schema and run the prompt against a small, hand-labeled dataset to calibrate drift score thresholds against human judgment.

This prompt is designed to run as a post-deployment analysis step inside a canary release pipeline, not as a real-time user-facing call. After routing a percentage of traffic to the new prompt version, collect a statistically meaningful sample of outputs from both the baseline and canary. Feed paired outputs into this prompt along with the evaluation rubric and deployment thresholds. The prompt returns a structured diff report, not a binary pass/fail. Your harness must parse the JSON output and enforce the promotion/rollback decision based on the recommendation field and your pre-defined gate rules.

Wiring the harness: Construct the [BASELINE_OUTPUTS] and [CANARY_OUTPUTS] arrays by sampling the same inputs across both versions. Each entry should include the input, the model response, and any relevant metadata (latency, token count, tool calls). The [EVALUATION_CRITERIA] block should be a JSON array of weighted dimensions—such as accuracy, tone, refusal rate, and format compliance—with acceptable drift thresholds. The [DEPLOYMENT_THRESHOLDS] block defines your gate: maximum allowed semantic drift score, maximum regression count, and whether a HOLD recommendation should block promotion or just flag for review. Use a structured output API (e.g., response_format with a JSON schema) to enforce the DiffReport schema. If the model returns malformed JSON, retry once with the raw output and a repair instruction. If the second attempt fails, escalate to a human reviewer and mark the gate as HOLD.

Validation and logging: Before acting on the recommendation, validate that semantic_drift_score is a float between 0 and 1, regression_flags is an array of objects with required severity and description fields, and recommendation is one of PROMOTE, HOLD, or ROLLBACK. Log the full prompt input, raw output, parsed report, and validation result to your deployment audit trail. If the canary sample size is below your statistical significance threshold, the harness should reject the comparison and request more data rather than making a low-confidence decision. For high-risk domains, always require a human to acknowledge a ROLLBACK recommendation before the pipeline executes it. Wire the final decision into your deployment orchestrator—if ROLLBACK, shift all traffic back to baseline; if HOLD, pause the canary at the current percentage and alert the release manager; if PROMOTE, proceed to the next ramp step.