Canary Deployment Health Evaluation Prompt

This prompt is for Site Reliability Engineers (SREs) and release coordinators who need to make a data-driven decision about a canary deployment. The job-to-be-done is straightforward: compare the health of a canary instance against a stable control baseline using key metrics like error rates, latency distributions, and business KPIs, then produce a clear recommendation to promote, extend the observation window, or roll back. The prompt assumes you already have access to structured metric data from your observability stack (e.g., Datadog, Prometheus, Grafana) and that you can provide it as part of the input context. It is not a replacement for your monitoring dashboard; it is a reasoning layer that synthesizes the numbers into a defensible operational decision.

Use this prompt when the decision is too nuanced for a simple threshold alert. For example, a 1% error rate increase might be statistically insignificant or a sign of a systemic failure depending on the baseline, traffic volume, and business impact. The prompt is designed to weigh these factors and flag when the evidence is insufficient to make a confident call. Do not use this prompt for real-time, sub-second alerting where an automated circuit breaker is already in place. It is also not suitable for evaluating infrastructure-level changes (e.g., kernel patches, cluster scaling) unless those changes manifest in the application-level metrics you provide. The ideal user is an on-call engineer or release manager who understands the metrics but needs a structured, unbiased second opinion under time pressure.

Before using this prompt, ensure you have a clean, structured representation of your canary and control metrics over a meaningful observation window. The prompt requires explicit statistical context—sample sizes, p-values, or confidence intervals—to avoid over-indexing on noise. If your observability pipeline cannot provide this, the prompt will correctly flag low confidence rather than fabricate certainty. A critical constraint is that the final decision to promote or roll back must always involve a human approval step, especially for high-risk services. The prompt's output is a recommendation with evidence, not an automated action. Wire it into a release runbook where the SRE reviews the evaluation before executing the next step.

Avoid using this prompt for canary evaluations where the control and canary are running fundamentally different hardware, configurations, or data sets that make a direct comparison invalid. The prompt will detect and flag such confounders if you include them in the context, but it cannot correct for a broken experimental design. For multi-service canary deployments, evaluate each service independently and then use a separate dependency readiness prompt to synthesize the cross-service picture. Finally, treat the prompt's statistical confidence notes as advisory. If it says 'low confidence due to insufficient sample size,' extend the observation window rather than overriding the recommendation with gut feeling.

This prompt template is designed to be wired directly into your canary analysis pipeline. It expects structured metric inputs—error rates, latency percentiles, business KPIs—for both the canary and control groups over a defined observation window. The model's job is not to collect metrics but to reason about their divergence, assess statistical significance, and produce a clear recommendation with confidence notes. Use this template as the core instruction block in your deployment gate, feeding it pre-computed metric summaries from your observability stack.

textCopy
You are a canary deployment health evaluator for an SRE team. Your task is to compare canary metrics against baseline control metrics and recommend whether to promote the canary, extend the observation window, or roll back immediately.

## INPUTS
- Canary Metrics: [CANARY_METRICS]
- Control Metrics: [CONTROL_METRICS]
- Observation Window: [OBSERVATION_WINDOW]
- SLO Thresholds: [SLO_THRESHOLDS]
- Deployment Context: [DEPLOYMENT_CONTEXT]

## OUTPUT SCHEMA
Return a JSON object with the following fields:
{
  "recommendation": "promote" | "extend_observation" | "rollback",
  "confidence": "high" | "medium" | "low",
  "summary": "One-sentence rationale for the recommendation.",
  "metric_analysis": [
    {
      "metric_name": "string",
      "canary_value": "number",
      "control_value": "number",
      "deviation_percent": "number",
      "direction": "increased" | "decreased" | "unchanged",
      "breaches_slo": "boolean",
      "statistical_significance": "significant" | "borderline" | "not_significant",
      "assessment": "Brief interpretation of this metric's signal."
    }
  ],
  "risk_factors": ["List of specific risks identified in the canary behavior."],
  "conditions_for_promotion": ["If recommendation is extend_observation, list what must normalize before promotion."],
  "rollback_triggers": ["If recommendation is promote, list conditions that would trigger an immediate rollback after promotion."]
}

## CONSTRAINTS
- Do not recommend promotion if any SLO-breaching metric shows statistical significance.
- If multiple metrics show borderline significance in the same negative direction, treat the pattern as a risk signal and lean toward extend_observation.
- If any metric shows a deviation greater than [CRITICAL_DEVIATION_THRESHOLD], recommend rollback regardless of statistical significance.
- If the observation window is less than [MINIMUM_OBSERVATION_MINUTES], note low confidence due to insufficient data.
- Always include specific metric names and values in your analysis. Do not generalize.
- If business metrics diverge negatively while technical metrics remain stable, flag this as a silent failure risk.

## EVALUATION STEPS
1. For each metric, calculate the percentage deviation between canary and control.
2. Check each metric against its SLO threshold.
3. Assess the direction and magnitude of deviation.
4. Consider whether deviations correlate across metrics (e.g., error rate and latency rising together).
5. Weigh the deployment context: is this a low-traffic canary, a high-risk change, or during a peak traffic period?
6. Form a recommendation based on the preponderance of evidence, not a single metric.
7. Assign confidence based on observation duration, traffic volume, and signal clarity.

To adapt this template for your environment, replace each square-bracket placeholder with data from your observability pipeline. [CANARY_METRICS] and [CONTROL_METRICS] should be structured as arrays of metric objects with name, value, and unit fields. [SLO_THRESHOLDS] should map metric names to their acceptable ranges. [CRITICAL_DEVIATION_THRESHOLD] and [MINIMUM_OBSERVATION_MINUTES] are tunable parameters that should be set based on your service's baseline stability and traffic patterns. If your canary system uses automated anomaly detection, include those anomaly scores as additional metrics rather than relying on the model to detect anomalies from raw data. Always validate the output JSON against the schema before acting on the recommendation—a malformed response should trigger a retry, not a deployment decision.

This prompt is designed to sit inside an automated canary analysis stage of your deployment pipeline, not as a standalone chatbot. The typical integration point is after a canary has been running for a defined observation window (e.g., 10–30 minutes) and metric data is available from your observability stack. The harness should fetch baseline and canary metrics from your time-series database (Prometheus, Datadog, CloudWatch), format them into the [BASELINE_METRICS] and [CANARY_METRICS] placeholders, and inject any relevant [ANOMALY_DETECTION_ALERTS] before calling the model. The prompt expects structured metric input—raw log lines or unstructured dashboards will produce unreliable evaluations.

Validation and retry logic is critical because this prompt drives a deployment decision. After receiving the model's JSON output, validate the recommendation field against the allowed enum values (promote, extend_observation, rollback). If the output fails to parse or contains an invalid recommendation, retry once with a stricter schema reminder appended to the prompt. If the second attempt also fails, escalate to a human operator and default to extend_observation as the safe fallback. Log every evaluation attempt—including the raw prompt, model response, parse success/failure, and final recommendation—to your deployment audit trail. For high-risk services, add a human approval gate: if the recommendation is rollback or if the confidence score falls below a configurable threshold (e.g., 0.7), route the evaluation to an on-call SRE for manual review before the pipeline acts on it.

Model choice matters. This prompt requires strong numerical reasoning and structured output discipline. Use a model with proven JSON mode or function-calling reliability (GPT-4o, Claude 3.5 Sonnet, or equivalent). Avoid smaller or older models that may hallucinate statistical interpretations or conflate correlation with causation. Set temperature=0 to maximize deterministic output. If your deployment pipeline has strict latency requirements, consider pre-warming the model endpoint or using a dedicated inference instance. Do not use this prompt with a general-purpose chat interface—always call it through an API with structured output parsing. Tool integration is recommended: provide a fetch_metric_history tool so the model can request additional data if the supplied metrics are insufficient, rather than guessing. Finally, test this harness against historical canary deployments where you know the correct outcome (promote, rollback, or extend) to calibrate thresholds and catch evaluation drift before it affects a live production decision.