Canary Release Observability Design Prompt

This prompt is for observability engineers and SREs who need to design the monitoring and alerting surface for a canary release before it ships. The job-to-be-done is producing a concrete, reviewable observability design that connects traffic shaping to specific metrics, anomaly detection rules, statistical significance thresholds, and dashboard layouts. The ideal user already understands the service architecture, its key health indicators (RED metrics, business KPIs), and the canary's traffic routing mechanism. They are not starting from zero; they need a structured thinking partner to pressure-test their signal selection and ensure no blind spots exist between the load balancer config and the on-call pager.

You should use this prompt when the canary's blast radius, traffic percentage, and evaluation window are defined but the observability wiring is still a sketch. It is appropriate for greenfield canary designs and for auditing existing deployments where false-positive alerts or late-detection failures have eroded trust in the canary process. The prompt forces explicit reasoning about metric cardinality (avoiding high-cardinality labels that explode time-series DB costs), alert sensitivity tuning (separating regression detection from noise), and the integration point between your metrics store and the automated rollback controller. Do not use this prompt for simple health-check pings, basic uptime monitoring, or choosing a vendor. It is not a replacement for load testing or chaos engineering; it designs the lenses through which you will observe those activities.

Before running this prompt, gather the service's SLI/SLO definitions, the canary's traffic step-up schedule, and a list of existing dashboards and alerts. The prompt will ask you to reason about statistical tests (e.g., t-tests, Mann-Whitney U) and their minimum sample sizes, so have your expected request volume and latency distributions handy. The output is a design document, not a Terraform module. After you receive the design, you must implement the metrics, dashboards, and alert rules in your observability stack, then validate them against a simulated canary regression. The prompt's greatest risk is producing a theoretically sound design that fails on real data because baseline metrics are noisy or the anomaly detection model hasn't been calibrated. Always budget time for a dry-run canary with known-good and known-bad versions to tune thresholds before relying on the design for production traffic.

The following prompt template is designed to be copied directly into your AI harness. It uses square-bracket placeholders for all dynamic inputs, ensuring you can adapt it to your specific service, metrics infrastructure, and risk tolerance. The prompt instructs the model to act as an observability architect and produce a structured design review, not just a list of metrics. Before using it, gather your service's SLI definitions, your current monitoring stack's capabilities, and the specific traffic-shaping mechanism you plan to use for the canary.

textCopy
You are an observability architect specializing in safe release engineering. Your task is to design the complete observability surface for a canary release of the service described below. Do not just list metrics; produce a design that connects metrics to automated decision-making.

## Service Context
- Service Name: [SERVICE_NAME]
- Service Function: [SERVICE_DESCRIPTION]
- Canary Traffic Percentage: [CANARY_PERCENTAGE]%
- Baseline (Stable) Version: [BASELINE_VERSION]
- Canary Version: [CANARY_VERSION]
- Deployment Environment: [ENVIRONMENT]

## Existing Monitoring Stack
- Metrics Backend: [METRICS_BACKEND, e.g., Prometheus, Datadog]
- Logging Backend: [LOGGING_BACKEND]
- Tracing Backend: [TRACING_BACKEND]
- Alerting Platform: [ALERTING_PLATFORM]

## Design Constraints
- [CONSTRAINT_1, e.g., Must not increase metric cardinality by more than 10%]
- [CONSTRAINT_2, e.g., All alerts must resolve within 5 minutes of a rollback]
- [CONSTRAINT_3, e.g., Observability logic must not add more than 1ms p99 latency]

## Required Output Sections
1.  **Key Health Signals (The Golden Signals):** Identify the 3-5 most critical metrics for this specific service (e.g., latency p95, error rate, throughput, saturation). Justify each choice against the service function.
2.  **Canary vs. Baseline Comparison Logic:** Define the exact statistical method for comparing signals between the canary and baseline cohorts (e.g., two-sided t-test, Mann-Whitney U test). Specify the p-value threshold for significance and the minimum sample size before a comparison is considered valid.
3.  **Anomaly Detection Integration:** Describe how you would integrate this comparison into an anomaly detection framework. Should it be a simple threshold, a dynamic baseline, or a seasonal model? Justify your choice.
4.  **Dashboard Composition:** Design a purpose-built canary dashboard. Specify the exact charts, their axes, and the critical overlays (e.g., deployment markers, baseline bands). The dashboard should answer "Is the canary healthy?" in under 5 seconds.
5.  **Alert Sensitivity and Tuning:** Define the alerting rules. What conditions trigger a warning vs. a critical alert? How do you prevent alert fatigue from noisy, low-sensitivity metrics? Include a specific plan for tuning `alert_sensitivity` during the first hour of the canary.
6.  **Metric Cardinality Risk Assessment:** Analyze the proposed design for metric cardinality explosion. Identify any high-cardinality labels (like user ID or session ID) and propose safe alternatives or aggregation strategies.

## Output Format
Produce a single, valid JSON object with the following schema:
{
  "design_title": "string",
  "golden_signals": [{"metric_name": "string", "justification": "string", "query": "string"}],
  "comparison_logic": {"statistical_method": "string", "p_value_threshold": "number", "min_sample_size": "number"},
  "anomaly_detection": {"method": "string", "justification": "string"},
  "dashboard": {"description": "string", "panels": [{"title": "string", "type": "string", "queries": ["string"]}]},
  "alerting_rules": [{"name": "string", "condition": "string", "severity": "warning | critical", "tuning_plan": "string"}],
  "cardinality_risk_assessment": {"high_cardinality_labels": ["string"], "mitigation_strategies": ["string"]}
}

To adapt this template, replace the placeholders in the Service Context and Existing Monitoring Stack sections with your specific details. The Design Constraints section is critical for grounding the model's output in your operational reality; if you have no explicit constraints, replace them with standard ones like "must not degrade dashboard load time" or "must use existing tooling." The output schema is strict JSON to allow for direct parsing by a release automation controller. If your release process requires a human-in-the-loop, add a final instruction: "Before finalizing, add a 'Human Review Notes' field to the JSON output and flag any recommendation with a confidence score below 0.8."

This prompt is designed to be called programmatically as part of a release review pipeline, not as a one-off chat interaction. The implementation harness should treat the prompt as a structured design review function: it receives a canary release specification as input and returns a validated observability design assessment. The harness is responsible for supplying the correct context, enforcing the output schema, validating the response, and deciding whether the result requires human review before the canary proceeds. Because canary releases directly affect production traffic, the harness must default to conservative behavior—blocking the release pipeline on validation failures or high-severity findings unless explicitly overridden by an authorized reviewer.

Wire the prompt into your release orchestration system (e.g., a CI/CD pipeline, a deployment gate service, or an internal review tool) with the following components. Input assembly: collect the canary specification including traffic shaping rules, target metrics, anomaly detection configuration, statistical significance thresholds, dashboard composition, and alert sensitivity settings. Map these into the [CANARY_SPEC] placeholder. Model selection: use a model with strong reasoning capabilities and structured output support (GPT-4o, Claude 3.5 Sonnet, or equivalent). Enable JSON mode or structured outputs to enforce the [OUTPUT_SCHEMA]. Validation layer: after receiving the model response, validate it against the expected schema—check that all required fields are present, severity scores are within range, and findings include specific references to the input specification. Reject responses that contain hallucinations (e.g., referencing metrics or thresholds not present in the input). Retry logic: on validation failure, retry up to two times with the validation errors appended to the prompt as additional [CONSTRAINTS]. If all retries fail, escalate to a human reviewer with the partial output and error context. Human review gate: route any assessment containing severity: critical findings or confidence: low scores to a release review queue. The harness should block canary progression until a designated reviewer acknowledges or overrides the findings. Logging: record the full prompt, response, validation results, retry count, and reviewer decision for auditability. This trace is essential for post-incident review and for improving the prompt over time.

Avoid wiring this prompt directly into an automated canary abort mechanism without human review. The model can identify observability gaps and suggest improvements, but it cannot guarantee that all risks are captured. The harness should treat the prompt output as a design review artifact that informs a human decision, not as an autonomous gate. When integrating with observability platforms (e.g., Datadog, Prometheus, Grafana), do not pass model-generated thresholds or queries directly into production monitoring without manual verification—metric cardinality errors or misconfigured anomaly detection can generate false positives that undermine trust in the canary process. Start with a shadow mode deployment where the prompt runs alongside existing review processes, compare results for three to five release cycles, and only then promote it to a blocking gate with the human review safeguards described above.