Chaos Engineering Observability Validation Prompt

Use this prompt when you are an SRE or platform engineer preparing to run a chaos engineering experiment and you need to validate that your observability stack will surface the expected failure signals before you inject the fault. The prompt takes your experiment hypothesis, the target service architecture, and your current observability configuration as input, then produces a structured validation checklist. This checklist maps each hypothesis to the specific metrics, logs, traces, dashboards, and alerts that should fire if observability is working correctly. The core job is pre-experiment verification: confirming that you can see the failure before you cause it, which prevents the worst-case outcome of running a chaos experiment that succeeds in breaking the system but fails to generate any actionable signal.

The ideal user has already defined a chaos experiment hypothesis (for example, 'if we terminate 50% of payment-service pods, the checkout API should show elevated latency but zero errors due to retry logic') and has access to their observability platform's configuration. You need to provide the experiment hypothesis, a description of the target service and its dependencies, your current dashboard and alert definitions, and any known observability gaps. The prompt works best when the hypothesis is specific and falsifiable. Avoid using this prompt for experiments where the hypothesis is vague ('see what happens') or where the observability stack is not yet deployed, because the output will be a list of gaps rather than a validation checklist.

Do not use this prompt as a substitute for running the experiment itself. It validates observability readiness, not system behavior. It also should not be used for post-incident review of observability gaps after a real outage, because the inputs and evaluation criteria differ. For post-incident analysis, use the Incident Retrospective Observability Gap Analysis Prompt instead. If your experiment involves security-sensitive failure modes such as authentication bypass or encryption degradation, add a human review step before executing the experiment, because the prompt may not catch signals that require security-specific telemetry. The output is a planning document, not a replacement for judgment about whether the experiment is safe to run in production.

This prompt template is designed to produce a validation checklist that maps each chaos experiment hypothesis to the specific observability signals—dashboards, metrics, logs, traces, and alerts—that should surface the failure. Use it before running an experiment to confirm that your observability stack will actually detect the injected fault. The template assumes you have already defined your experiment hypotheses and have access to your service's observability architecture. If you are still designing your experiment, start with an experiment design prompt first.

textCopy
You are an SRE observability validator. Your task is to produce a validation checklist that confirms whether the current observability stack will surface the failures injected by a chaos experiment.

## EXPERIMENT CONTEXT
[EXPERIMENT_CONTEXT]

## HYPOTHESES
[HYPOTHESES]

## OBSERVABILITY ARCHITECTURE
[OBSERVABILITY_ARCHITECTURE]

## CONSTRAINTS
[CONSTRAINTS]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "experiment_id": "string",
  "validation_date": "string (ISO 8601)",
  "hypothesis_validations": [
    {
      "hypothesis_id": "string",
      "hypothesis_description": "string",
      "expected_signals": [
        {
          "signal_type": "metric | log | trace | alert | dashboard",
          "signal_name": "string",
          "expected_behavior": "string",
          "detection_path": "string (e.g., 'Grafana dashboard X panel Y', 'Alert rule Z in Prometheus')",
          "current_coverage": "covered | partial | missing",
          "gap_description": "string (if not covered, what is missing)"
        }
      ],
      "overall_assessment": "validated | partial_gap | critical_gap",
      "recommendation": "string (what to add or change before running the experiment)"
    }
  ],
  "blind_spot_summary": [
    {
      "gap_category": "string (e.g., 'missing alert', 'insufficient log detail', 'trace sampling gap')",
      "affected_hypotheses": ["hypothesis_id"],
      "severity": "critical | high | medium | low",
      "remediation": "string"
    }
  ],
  "pre_experiment_checklist": [
    "string (actionable item to complete before experiment execution)"
  ],
  "overall_readiness": "ready | conditional | not_ready",
  "human_review_required": true
}

## INSTRUCTIONS
1. For each hypothesis, identify every signal type that should detect the failure.
2. Cross-reference expected signals against the provided observability architecture.
3. Flag any signal that is missing, misconfigured, or sampled out.
4. Produce a concrete, actionable pre-experiment checklist.
5. If any critical gap exists, set overall_readiness to "not_ready" and require human review.
6. Do not invent signals or dashboards not described in the observability architecture.
7. If the observability architecture is incomplete, note that as a blind spot.

Adapt this template by replacing each square-bracket placeholder with concrete data from your chaos engineering workflow. [EXPERIMENT_CONTEXT] should include the service name, experiment type (e.g., network partition, pod kill, latency injection), blast radius, and steady-state definition. [HYPOTHESES] should list each formal hypothesis with its unique ID and description. [OBSERVABILITY_ARCHITECTURE] should describe your actual dashboards, alert rules, log pipelines, trace sampling rates, and metric names—the more specific, the better the gap analysis. [CONSTRAINTS] can include regulatory requirements, data sensitivity rules, or organizational policies that affect what can be monitored. After copying the template, test it against a known past incident to see if the prompt correctly identifies gaps that were missed at the time.

Before using this prompt in production, validate its output against a human-reviewed baseline. Run it on three experiments: one where observability is known to be complete, one with known gaps, and one with unknown gaps. Compare the prompt's gap detection against the human assessment. If the prompt misses a gap a human found, add that gap as a counterexample in a few-shot section. If the prompt hallucinates signals that do not exist, tighten the instruction to require evidence from the provided architecture. For high-severity experiments in production, always require human review of the validation checklist before experiment execution—this prompt is a diagnostic aid, not a replacement for SRE judgment.

This prompt is designed to be used as a pre-flight check before executing a chaos experiment, or as a post-mortem analysis tool after an experiment has concluded. In an application harness, you should invoke it with a structured input payload containing the experiment hypothesis, the planned fault injection method, the blast radius, and the current state of your observability stack. The model's output is a structured validation checklist, so your harness must parse the JSON output strictly and validate it against a known schema before surfacing it to the SRE. A failed parse or a missing required field should trigger a retry with a more explicit [OUTPUT_SCHEMA] constraint, not a silent fallback.

To wire this into a chaos engineering platform like Gremlin, Chaos Mesh, or LitmusChaos, build a pre-experiment hook that calls your LLM endpoint with the experiment definition. The harness should construct the [EXPERIMENT_HYPOTHESIS], [FAULT_TYPE], [BLAST_RADIUS], and [OBSERVABILITY_STACK] variables from the experiment configuration. After receiving the model's checklist, run a programmatic validator that checks for: (1) each hypothesis is mapped to at least one specific dashboard panel or log query, (2) every alert mentioned has a defined threshold and evaluation window, and (3) no checklist item is a generic placeholder like 'check dashboards.' If the validator fails, append the specific failure reason to the [CONSTRAINTS] field and retry once. Log the full prompt, response, and validation result to your experiment audit trail for post-incident review.

For high-severity experiments in production, do not rely solely on the model's output. The harness should flag any checklist item where the model's confidence is implicitly low—detectable by vague language like 'consider checking' or 'might be useful'—and route those items to a human SRE for manual enrichment. Implement a review queue in your experiment workflow that blocks the chaos injection until all flagged items are resolved. The model's output should be treated as a draft that accelerates human review, not as a final authorization to proceed. After the experiment, feed the actual observability signals back into a follow-up prompt to generate a gap analysis, closing the loop between expected and actual detection.