Observability and Logging Adequacy Charter Prompt

This prompt is for SRE-minded testers and platform engineers who need to verify that when a system fails, it fails informatively. The core job-to-be-done is generating a time-boxed exploratory test charter that targets the quality of diagnostic signals—logs, metrics, traces, and alerts—under induced failure conditions. The ideal user is someone who understands the system's architecture and failure domains but needs a structured, mission-driven plan to probe whether the current observability stack provides enough context for an on-call engineer to diagnose root cause without resorting to guesswork or direct shell access. Use this prompt during pre-release reliability testing, observability readiness reviews, or chaos engineering preparation. It is not a replacement for automated monitoring checks, real incident response drills, or postmortems; it is a pre-flight check on the telemetry itself.

The prompt requires specific input context to be effective: a description of the system or component under test, the failure mode to be induced (e.g., a database connection timeout, a dependent service returning 503s, a message queue backlog), and the available observability tools (e.g., Elasticsearch, Datadog, Jaeger, Prometheus/Alertmanager). The output is a structured charter containing a time-boxed mission, specific investigation targets (e.g., 'Verify that the error log includes the correlation ID from the inbound request'), oracle heuristics (e.g., 'The trace should show a clear span attributing the 503 to the downstream call, not a generic 500'), and a pass/fail standard centered on diagnosticability. The charter is designed to be executed immediately by a human tester who will inject the failure and then explore the resulting telemetry.

Do not use this prompt when the system has no observable surface (e.g., a black-box third-party service with no access to internals), when the failure mode is purely cosmetic and generates no telemetry, or when the goal is to test the correctness of business logic rather than the quality of its failure signals. This prompt also assumes a baseline level of observability infrastructure; if your system lacks structured logging, distributed tracing, or alerting rules, the charter will correctly identify those gaps as failures, but the exercise may be more of an inventory of missing capabilities than a diagnostic drill. For high-risk regulated systems, ensure that any induced failures are conducted in pre-production environments and that the resulting charter includes a human review step before execution to prevent accidental impact on shared staging infrastructure.

This prompt template produces a time-boxed exploratory test charter for SRE-minded testers investigating whether failure modes generate sufficient diagnostic signals. Replace the square-bracket placeholders with your specific system context, failure scenarios, and observability stack details. The charter output should give a tester a clear mission, investigation targets, and oracle heuristics for judging log completeness, alert triggering, trace continuity, and error context quality.

textCopy
You are an exploratory test charter generator specializing in observability and logging adequacy. Generate a time-boxed exploratory test charter that investigates whether the system under test produces sufficient diagnostic signals when failures occur.

# REQUIRED INPUTS
- System or component under test: [SYSTEM_UNDER_TEST]
- Failure scenarios to probe: [FAILURE_SCENARIOS]
- Observability stack in use: [OBSERVABILITY_STACK]
- Expected diagnostic signals: [EXPECTED_SIGNALS]
- Session time box: [SESSION_DURATION]
- Risk level of missed signals: [RISK_LEVEL]

# CHARTER STRUCTURE
Generate a charter with these sections:
1. **Mission Statement**: One sentence describing what diagnostic capability is being investigated and why it matters.
2. **Investigation Targets**: Specific logs, metrics, traces, alerts, or error payloads to inspect during the session.
3. **Failure Induction Plan**: How to safely trigger or simulate each failure scenario in a test environment.
4. **Oracle Heuristics**: For each failure scenario, describe what a well-instrumented system should produce (log entries, alert firings, trace spans, error context fields).
5. **Data Capture Instructions**: What to record during the session—screenshots of dashboards, exported log lines, trace waterfall captures, alert notification timestamps.
6. **Adequacy Rubric**: A 3-level scale (Insufficient / Adequate / Excellent) for judging whether the diagnostic output would allow a hypothetical on-call engineer to diagnose the root cause within 15 minutes.

# CONSTRAINTS
- Do not propose destructive failure induction in production environments.
- Include explicit safety checks before each failure induction step.
- Flag any scenario where diagnostic gaps could delay incident response beyond acceptable thresholds.
- If [RISK_LEVEL] is high, include a recommendation for human review of the charter before execution.

After pasting the template, fill each placeholder with concrete details. For [FAILURE_SCENARIOS], list specific conditions such as database connection pool exhaustion, upstream API timeout, or malformed request payloads. For [EXPECTED_SIGNALS], specify what your observability stack should emit—structured log fields, trace attributes, alert labels, or dashboard metrics. If the [RISK_LEVEL] is high, the generated charter should include a pre-execution review checkpoint and recommend pairing testers with an on-call engineer during the session. Validate the output by checking that every failure scenario maps to at least one oracle heuristic and that the adequacy rubric has clear, distinguishable levels.

This prompt is designed to be invoked programmatically as part of a chaos engineering or failure-injection testing pipeline. It is not a one-off chat interaction; it should be triggered automatically when a new failure mode is introduced in a staging environment, when a service's logging configuration changes, or when an incident postmortem identifies a diagnostic gap. The harness must supply the prompt with structured context about the system under test, the specific failure being induced, and the expected observability surface. Treat the prompt output as a test plan artifact that must be versioned, reviewed, and tracked alongside the failure test results.

To integrate this into a CI/CD pipeline, build a test harness that: (1) reads a failure scenario definition from a configuration file or incident ticket; (2) populates the [SYSTEM_CONTEXT], [FAILURE_MODE], and [OBSERVABILITY_SURFACE] placeholders with structured data from your service catalog, architecture diagrams, and monitoring tool configurations; (3) calls the LLM with the completed prompt; (4) validates the output schema—each charter must include a mission_statement, investigation_targets array, expected_signals object, and diagnosability_criteria; (5) stores the generated charter as a JSON artifact in your test management system; and (6) gates the pipeline if the charter fails to meet minimum specificity thresholds (e.g., fewer than three investigation targets or missing signal expectations for critical log streams). Use a model with strong instruction-following and structured output support, such as gpt-4o or claude-3.5-sonnet, with response_format set to json_schema where available. Implement a retry loop with exponential backoff for schema validation failures, but cap retries at three attempts before flagging for human review.

The highest-risk failure mode for this harness is producing a charter that looks structurally valid but contains vague, untestable investigation targets. To guard against this, add an automated evaluation step after generation. Use a secondary LLM judge prompt that scores each charter on: target specificity (are investigation targets concrete actions an SRE can execute?), signal completeness (do expected signals cover logs, metrics, traces, and alerts?), and diagnosability (would an on-call engineer have enough information to diagnose the root cause within 15 minutes?). Set a minimum score threshold for each dimension. Charters that pass automated validation should still be spot-checked by a reliability engineer during the first five pipeline runs to calibrate the eval rubric. Never ship a generated charter directly into a production chaos experiment without human acknowledgment when the failure mode involves data loss, security boundaries, or revenue-critical paths.