Canary Deployment Evaluation Prompt for Prompts

This prompt is designed for DevOps engineers and AI platform teams who need to evaluate a new prompt version (the canary) against a stable baseline during a staged rollout. Instead of relying solely on numeric metric dashboards, this prompt acts as an analytical co-pilot. It ingests structured comparison data from your traffic-split harness and produces a reasoned canary analysis report. The report includes statistical significance checks, regression detection, and a clear promote/rollback recommendation. Use this when you have a canary deployment running with a defined traffic split and have collected metric data from both the baseline and canary prompt variants. It is not a replacement for your metric collection pipeline; it is the decision-support layer that interprets the results.

The ideal user is an engineer or release manager who already has a monitoring harness in place—such as a CI/CD pipeline with a traffic-splitting gateway—and needs to make a data-informed promotion decision. You must supply the prompt with structured metric summaries, including sample sizes, mean values, variance, and any predefined safety or quality thresholds. The prompt does not perform real-time metric collection or connect to your observability stack; it expects pre-aggregated data as input. This separation keeps the prompt focused on analysis and recommendation rather than data plumbing.

Do not use this prompt as a substitute for a statistical package or for making decisions on extremely small sample sizes where confidence intervals are unreliable. It is also not suitable for evaluating prompts in isolation without a live baseline comparison. If you need to define the acceptance criteria themselves—such as what constitutes a regression or what latency budget is acceptable—use the Staging-to-Production Acceptance Criteria Prompt or the Rollback Trigger Definition Prompt first. Once those gates are defined and metric data is flowing, this prompt helps you interpret the results and decide whether to promote or roll back.

The following prompt template is designed to be dropped into your canary evaluation harness. It expects structured input data representing the performance of a baseline and canary prompt version across a defined traffic split. The model's job is to act as an automated release analyst, comparing the two datasets against your predefined thresholds and producing a clear, evidence-backed recommendation. The output is a strict JSON schema suitable for consumption by a CI/CD pipeline or deployment automation tool.

textCopy
You are a release automation analyst evaluating a canary deployment for an AI prompt. Your task is to compare the performance of a BASELINE prompt version against a CANARY prompt version using the provided data and thresholds. You must produce a structured JSON report with a clear PROMOTE or ROLLBACK recommendation.

## INPUT DATA

### Traffic Configuration
- Traffic Split: [TRAFFIC_SPLIT_PERCENTAGE]% canary
- Evaluation Window: [EVALUATION_WINDOW_MINUTES] minutes
- Minimum Sample Size Required: [MIN_SAMPLE_SIZE]

### Baseline Metrics
```json
[BASELINE_METRICS_JSON]

Canary Metrics

jsonCopy
[CANARY_METRICS_JSON]

Threshold Configuration

jsonCopy
[THRESHOLD_CONFIGURATION_JSON]

EVALUATION CRITERIA

1. Sample Size Check: Verify that the canary sample size meets or exceeds the minimum required. If not, recommend "INCONCLUSIVE" with a clear explanation.
2. Metric Comparison: For each metric in the threshold configuration, compare the canary value against the baseline value and the defined threshold.
3. Regression Detection: Flag any metric where the canary performance is worse than the baseline by more than the allowed degradation threshold.
4. Statistical Significance: Note whether observed differences are likely meaningful given the sample size. Do not fabricate p-values; instead, describe the practical significance based on the magnitude of difference relative to the threshold.
5. Promotion Decision: Recommend "PROMOTE" only if ALL metrics pass their thresholds. Recommend "ROLLBACK" if ANY metric fails its threshold. Recommend "INCONCLUSIVE" if sample size is insufficient.

OUTPUT SCHEMA

You must respond with valid JSON matching this exact schema:

jsonCopy
{
  "recommendation": "PROMOTE | ROLLBACK | INCONCLUSIVE",
  "confidence": "HIGH | MEDIUM | LOW",
  "sample_size_check": {
    "canary_sample_size": <integer>,
    "minimum_required": <integer>,
    "sufficient": <boolean>
  },
  "metric_results": [
    {
      "metric_name": "<string>",
      "baseline_value": <number>,
      "canary_value": <number>,
      "threshold": <number>,
      "direction": "higher_is_better | lower_is_better",
      "absolute_change": <number>,
      "percentage_change": <number>,
      "passed": <boolean>,
      "severity": "PASS | WARNING | CRITICAL"
    }
  ],
  "regressions_detected": [
    {
      "metric_name": "<string>",
      "degradation_amount": <number>,
      "threshold_exceeded_by": <number>
    }
  ],
  "summary": "<A concise 2-3 sentence summary of findings and rationale for the recommendation.>",
  "risk_flags": ["<string>"]
}

CONSTRAINTS

• Do not invent or assume metric values not present in the input data.
• If the canary outperforms the baseline, note it but do not override a regression in another metric.
• If sample size is insufficient, set recommendation to "INCONCLUSIVE" regardless of metric values.
• The "summary" field must be self-contained and actionable for an on-call engineer.
• Flag any metric near its threshold (within 10%) as a risk flag even if it technically passes.
• Do not include markdown formatting inside JSON string values.

To adapt this template for your environment, replace the square-bracket placeholders with your actual data. The [BASELINE_METRICS_JSON] and [CANARY_METRICS_JSON] fields should contain your aggregated metric payloads—typically latency percentiles, error rates, token costs, quality scores from an LLM judge, or business metrics like task completion rates. The [THRESHOLD_CONFIGURATION_JSON] defines the acceptable bounds for each metric, including the direction of improvement and the maximum allowed degradation. Wire this prompt into your canary evaluation step so it runs automatically after the evaluation window closes, and route its JSON output directly to your deployment orchestrator. Always log the full prompt and response for auditability, and consider adding a human approval step if the confidence level is "LOW" or if any metric shows a "CRITICAL" severity regression.

The canary evaluation prompt is designed to sit inside a deployment pipeline that already controls traffic splitting between a baseline prompt version and a canary version. The harness must supply the prompt with structured metric data from both sides of the split, including sample sizes, raw counts, and distribution summaries. Without this data, the prompt cannot perform statistical significance checks or detect regressions. The harness should collect metrics from your observability stack—such as latency percentiles, error rates, token costs, and quality scores—and format them into the [BASELINE_METRICS] and [CANARY_METRICS] input blocks before invoking the model.

Validation occurs in two layers. First, the harness must validate the input data: confirm that sample sizes meet minimum thresholds for statistical power, verify that metric names match between baseline and canary, and reject runs where traffic split ratios fall outside the configured range (e.g., less than 5% canary traffic). Second, the harness must validate the model's output against the [OUTPUT_SCHEMA] before acting on the recommendation. Parse the promote_or_rollback field and check that it is exactly promote, rollback, or hold. If the model returns rollback, verify that at least one regression metric exceeds the configured severity threshold before triggering an automated rollback. For high-risk deployments, route hold and borderline promote decisions to a human approval queue with the full analysis report attached.

Model choice matters here. Use a model with strong quantitative reasoning and structured output reliability—GPT-4o, Claude 3.5 Sonnet, or equivalent. Avoid smaller models that may hallucinate statistical interpretations or miscalculate confidence intervals. Set temperature=0 to maximize output consistency across repeated evaluations. The prompt expects tool use for statistical computation; if your model provider does not support native function calling, pre-compute the statistical tests (t-test, chi-square, Mann-Whitney U as appropriate) in your harness code and pass the results into the prompt as pre-computed fields rather than asking the model to perform arithmetic. Log every evaluation result—including the full prompt, model response, parsed decision, and any human override—to an audit table keyed by prompt version and deployment ID. This trace is essential for post-incident review and for tuning the gate thresholds over time.

Retry logic should be conservative. If the model returns malformed JSON or fails schema validation, retry once with a stripped-down prompt that includes only the metric data and a terse instruction to return valid JSON. If the second attempt fails, fail closed: treat the evaluation as a hold and escalate to the on-call engineer. Do not retry more than twice, and never auto-promote on evaluation failure. Wire the harness into your CI/CD system as a blocking gate between the canary observation period and the full rollout step. The harness should emit a structured exit code or callback that your deployment orchestrator can consume: 0 for promote, 1 for rollback, 2 for hold or evaluation failure. This makes the prompt's decision machine-actionable without requiring a human to read the report before the pipeline can proceed.