Policy Update Canary Analysis Prompt

This prompt is designed for data scientists and AI operations engineers who must make an evidence-based decision on whether to promote, extend, or roll back a system prompt update. The primary job-to-be-done is converting raw split-traffic metrics from a canary deployment into a structured statistical analysis with a clear, defensible recommendation. Use this when you have already instrumented your application to collect side-by-side metrics from a control group (current system prompt) and a canary group (candidate system prompt) across key dimensions like safety refusal rates, task completion accuracy, latency, and output quality scores. The prompt assumes you are past the experiment design phase and are now at the decision gate, holding raw data that needs rigorous interpretation before a production change is authorized.

The ideal user has a working understanding of A/B testing methodology and can supply the required inputs: raw aggregate metrics for both groups, sample sizes, the evaluation period, and the pre-registered success criteria that were defined before the experiment began. You should also provide any known confounding factors, such as a traffic imbalance or an upstream model change that occurred mid-experiment. The prompt does not design the canary itself—it does not tell you how to split traffic, what metrics to track, or how long to run the experiment. It also does not connect to a live metrics database; you must extract and format the data before invoking the prompt. If you have not yet defined your evaluation criteria or lack sufficient sample sizes for statistical significance, do not use this prompt. Instead, return to experiment design and data collection before seeking a promotion decision.

Avoid using this prompt for low-risk, informal prompt changes where a full statistical analysis is overkill. If the policy update is a minor wording clarification with no measurable impact on safety or task performance, a lightweight manual review is more appropriate. Similarly, do not use this prompt when the canary and control groups are not truly comparable—for example, if the canary group received a different model version or if the traffic split was not random. The output includes confidence intervals and a recommendation, but the analysis is only as sound as the input data. Always pair the prompt's output with a human review step before executing a promotion or rollback in production, especially when safety metrics are involved. The next step after reading this section is to gather your raw metrics, format them according to the input schema, and proceed to the prompt template.

This template is designed to be pasted directly into your evaluation harness, LLM playground, or automated canary analysis pipeline. It accepts structured experiment data and produces a statistical comparison between canary and control groups, including a clear recommendation to promote, extend, or roll back the policy update. The prompt enforces confidence interval reporting, requires explicit handling of metric trade-offs, and refuses to make recommendations when data is insufficient.

textCopy
You are a statistical analysis engine evaluating a policy update canary experiment. Your output must be a strict JSON object following [OUTPUT_SCHEMA]. Do not include commentary outside the JSON.

## INPUT DATA
[EXPERIMENT_DATA]

## CONTEXT
- Policy version under test: [POLICY_VERSION_CANARY]
- Baseline policy version: [POLICY_VERSION_CONTROL]
- Experiment start: [EXPERIMENT_START_TIMESTAMP]
- Experiment end: [EXPERIMENT_END_TIMESTAMP]
- Traffic split: [TRAFFIC_SPLIT_PERCENTAGE]
- Minimum sample size required per metric: [MIN_SAMPLE_SIZE]
- Significance threshold (alpha): [ALPHA_THRESHOLD]
- Practical significance bounds (minimum effect size of interest): [EFFECT_SIZE_BOUNDS]

## CONSTRAINTS
1. For every metric, report the sample size, mean, standard deviation, confidence interval, p-value, and whether the result meets both statistical and practical significance.
2. If any metric has a sample size below [MIN_SAMPLE_SIZE], mark it as "INSUFFICIENT_DATA" and do not draw conclusions from it.
3. If safety metrics show statistically significant degradation at alpha=[ALPHA_THRESHOLD], the recommendation must be "ROLLBACK" regardless of other metric improvements.
4. If quality or performance metrics improve but safety metrics are within bounds, the recommendation may be "PROMOTE" or "EXTEND" based on confidence interval overlap with practical significance bounds.
5. If results are inconclusive (confidence intervals too wide, conflicting metric directions), recommend "EXTEND" with a required minimum additional sample size.
6. Explicitly flag any metric where the canary group underperforms the control group, even if the difference is not statistically significant.
7. Do not invent data. If [EXPERIMENT_DATA] is missing required fields, return an error object specifying which fields are absent.

## OUTPUT SCHEMA
{
  "experiment_summary": {
    "canary_version": "string",
    "control_version": "string",
    "duration_hours": number,
    "total_samples_canary": number,
    "total_samples_control": number,
    "traffic_split_actual": number
  },
  "metric_results": [
    {
      "metric_name": "string",
      "category": "SAFETY | QUALITY | PERFORMANCE",
      "canary": {
        "sample_size": number,
        "mean": number,
        "std_dev": number,
        "confidence_interval_95": [number, number]
      },
      "control": {
        "sample_size": number,
        "mean": number,
        "std_dev": number,
        "confidence_interval_95": [number, number]
      },
      "difference": number,
      "p_value": number,
      "statistically_significant": boolean,
      "practically_significant": boolean,
      "direction": "CANARY_BETTER | CONTROL_BETTER | NO_DIFFERENCE",
      "data_sufficiency": "SUFFICIENT | INSUFFICIENT_DATA"
    }
  ],
  "safety_assessment": {
    "any_safety_degradation": boolean,
    "degraded_metrics": ["string"],
    "confidence": "HIGH | MEDIUM | LOW"
  },
  "recommendation": {
    "action": "PROMOTE | EXTEND | ROLLBACK",
    "rationale": "string",
    "confidence_level": "HIGH | MEDIUM | LOW",
    "if_extend": {
      "additional_samples_needed": number,
      "estimated_days_to_complete": number
    },
    "if_rollback": {
      "primary_reason": "string",
      "secondary_concerns": ["string"]
    }
  },
  "caveats": ["string"]
}

## EXAMPLES
[EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

If [RISK_LEVEL] is "HIGH", add an additional field "human_review_required": true to the recommendation object and include a summary of findings suitable for a human reviewer in the "caveats" array.

To adapt this template, replace each square-bracket placeholder with your actual experiment data and configuration. The [EXPERIMENT_DATA] placeholder should contain a structured JSON array of per-metric observations from both canary and control groups. The [EXAMPLES] placeholder is optional but strongly recommended for high-stakes policy updates—include one or two few-shot examples showing correct output formatting for PROMOTE, EXTEND, and ROLLBACK scenarios. The [RISK_LEVEL] placeholder accepts "LOW", "MEDIUM", or "HIGH" and gates whether a human-review flag is injected into the output. Before running this in production, validate the output against the schema using a JSON schema validator in your pipeline, and log every recommendation alongside the input data for auditability. For regulated domains, always route ROLLBACK recommendations and any output where safety_assessment.any_safety_degradation is true through a human approval step before acting on the recommendation.

The Policy Update Canary Analysis Prompt is designed to be the final analytical step in an automated canary release pipeline, not a standalone chat interaction. It consumes structured experiment data produced by your metrics infrastructure and produces a machine-readable recommendation that gates the next stage of your deployment. The harness must enforce strict input validation, handle model failures gracefully, and ensure that a 'promote' decision is never made on malformed or incomplete data.

Wire the prompt into an application function that accepts a pre-formatted JSON payload containing canary_metrics, control_metrics, safety_thresholds, performance_thresholds, sample_sizes, and experiment_duration_hours. Before calling the model, validate that all required fields are present, numeric values are within sane ranges, and sample sizes exceed your minimum statistical power threshold. Use a structured output API call with a defined JSON schema for the response, including fields for recommendation (promote, extend, rollback), confidence_level, metric_summary, and risk_flags. Implement a retry loop with up to 3 attempts if the model returns invalid JSON or fails schema validation, and log every attempt with the raw input and output for auditability.

The most dangerous failure mode is a hallucinated statistical claim that passes schema validation but misrepresents the data. Mitigate this by implementing a post-generation validation layer that independently recalculates key statistics from the raw input data and compares them against the model's reported values. If the model's reported p-values, confidence intervals, or effect sizes deviate from your ground-truth calculations by more than a configurable tolerance, flag the output for human review and block automatic promotion. For high-stakes policy updates, route all 'promote' recommendations with confidence below 95% or any active risk_flags to a human approval queue before the deployment proceeds. Choose a model with strong quantitative reasoning capabilities for this task; smaller or older models frequently produce plausible-looking but incorrect statistical summaries. Never use this prompt as the sole gate for safety-critical policy changes without human review of both the input data quality and the model's reasoning.