Chaos Experiment Readiness

The readiness check must confirm that the observability stack is fully functional and that critical alerts are configured and routed correctly. This involves:

• Verifying metrics collection (e.g., Prometheus scrapes, Datadog agent)
• Ensuring dashboards reflect real-time system state
• Testing alert delivery via PagerDuty, Slack, or email to on-call engineers
• Validating that Service Level Indicator (SLI) dashboards are operational to measure experiment impact.

A validated, automated rollback procedure is the primary safety mechanism. The readiness check must test the rollback path without executing it. This includes:

• Verifying that circuit breakers or feature flags can be toggled to a safe state
• Ensuring deployment systems (e.g., ArgoCD, Spinnaker) can execute a rollback to a known-good version
• Confirming database migration rollback scripts are present and tested
• Testing the Dead Man's Switch or manual override procedure that engineers can trigger.

The system's ability to withstand failure depends on the health of its dependencies. The check must verify:

• Service Discovery Health: That the registry (Consul, etcd) is healthy and services are correctly registered.
• Quorum Readiness: For stateful services, that a majority of nodes are online to maintain consensus (e.g., Raft protocol health).
• Dependency Circuit Breakers: That clients for downstream services (databases, APIs) have functional circuit breakers to fail fast.
• Resource connections (database pools, message queues) are within healthy limits.

Chaos experiments often involve retries and partial failures. The system must be prepared to handle these safely.

• Idempotency Key Check: Validate that core operations (e.g., payments, orders) accept idempotency keys to prevent duplicate side-effects from retried requests.
• State Snapshot Integrity: If the experiment targets data layers, confirm that recent backups or snapshots are valid and restorable.
• Declarative State Verification: For Kubernetes or infrastructure-as-code, confirm the actual cluster state matches the declared manifests to avoid configuration drift during recovery.

A final pre-flight validation ensures the experiment is correctly constrained. This involves checking:

• The failure injection tool (e.g., Chaos Mesh, Gremlin) is configured to target only the approved namespace, host, or service.
• The experiment duration is set and that a time-based auto-abort is configured.
• The traffic routing (e.g., via a service mesh like Istio) ensures the experiment affects only a defined percentage of canary traffic, not all users.
• All manual approval gates in the experiment pipeline have been satisfied.

Human factors are critical. The readiness check confirms that the team is prepared to respond.

• The incident runbook for the specific failure scenario is accessible and up-to-date.
• Communication channels (war rooms, status pages) are prepared for activation.
• Key stakeholders are notified that an experiment is imminent.
• The team has conducted a pre-mortem or brief to anticipate potential failure modes of the experiment itself.

A predefined rule or condition that automatically initiates the reversion of a system to a previous known-good state upon detection of a critical failure. This is the primary safety mechanism validated during chaos experiment readiness. The trigger is typically based on:

• Service Level Objective (SLO) Violations: e.g., error rate > 1%, latency p99 > 500ms.
• Synthetic Transaction Failures: A key user journey breaks.
• Health Check Cascade: Critical dependencies become unhealthy.

Readiness involves verifying that the monitoring pipeline detecting the failure, the decision logic, and the deployment system's rollback capability are all integrated and functional.

A system design principle where functionality is reduced in a controlled, deliberate manner when a partial failure occurs, maintaining core operations while non-essential features are disabled. Chaos experiment readiness tests the degradation pathways.

• Core vs. Non-Core: The system must correctly identify which features can be turned off (e.g., product recommendations, avatar uploads) while keeping the primary service alive (e.g., login, core transaction processing).
• User Experience: Failures should be communicated clearly (e.g., 'Search is temporarily slow, here are recent items').
• Architectural Patterns: Implemented via feature flags, fallback caches, or default static responses.

Readiness validation confirms that when a key dependency fails, the system degrades according to design, rather than crashing completely.

A specific type of health check that verifies an application can successfully connect to and communicate with its external dependencies, such as databases, APIs, caches, or message queues. This is a fundamental readiness gate.

• Pre-Experiment Validation: All critical dependencies must pass their checks. Injecting chaos into a system already struggling with a latent dependency connection issue will produce invalid, noisy results.
• Depth: Checks should go beyond simple TCP connectivity to include authentication, a minimal read/write operation, or a schema version check.
• Integration: These checks are often part of the Readiness Probe in Kubernetes, ensuring a pod is not brought into service load balancers until its dependencies are confirmed alive.