Chaos Engineering

Every chaos experiment begins with a clear, falsifiable hypothesis about how the system should behave under stress. This transforms testing from random fault injection into a structured scientific inquiry.

• Example Hypothesis: "If the primary database node fails, the system's read API latency will remain under 200ms as traffic fails over to the replica."
• The experiment's value lies in proving or disproving this hypothesis, leading to concrete system improvements.

A cardinal rule of chaos engineering is to limit the potential impact, or blast radius, of an experiment. This ensures failures are contained and do not cause unacceptable user impact or business damage.

• Implementation Tactics:
  • Start experiments in a staging environment before production.
  • Target a small percentage of user traffic or a single, non-critical service.
  • Use feature flags and kill switches to abort experiments instantly.
• This principle enables safe experimentation in live systems.

Chaos engineering requires a quantifiable definition of a system's steady state—its normal, healthy operational behavior. Experiments measure deviations from this baseline.

• Steady-State Metrics often include:
  • Throughput (requests per second)
  • Error rate (percentage of failed requests)
  • Latency (p95, p99 response times)
• By continuously monitoring these Service Level Indicators (SLIs), engineers can objectively determine if an injected fault has caused unacceptable degradation.

To be effective, chaos experiments must be automated and integrated into the development lifecycle, not run as one-off, manual events. This is often called Continuous Verification.

• Integration Points:
  • CI/CD Pipelines: Run small-scope experiments on canary deployments before full release.
  • Scheduled "Game Days": Automatically execute experiments during off-peak hours.
  • This shifts resilience testing left and makes it a routine part of operations, constantly validating assumptions as the system evolves.

While testing in staging is a safe starting point, the highest-fidelity insights come from experiments in the production environment. Production contains unique complexities—real user traffic, data volumes, and hardware interactions—that cannot be fully simulated.

• Key Justifications:
  • Catches dependencies on production-only services or configurations.
  • Reveals how real user behavior interacts with failures.
  • Tests monitoring, alerting, and on-call response procedures under real conditions.
• This principle emphasizes that confidence is built by testing the system as it truly operates.

Effective chaos engineering explores a wide variability of potential failures, not just simple, predictable outages. Systems often fail in surprising, non-linear ways.

• Types of Failure Modes to Experiment With:
  • Latency Injection: Adding delays to network calls or disk I/O.
  • Resource Exhaustion: Saturating CPU, memory, or network bandwidth.
  • Partial Degradation: Returning corrupted data or specific HTTP error codes.
  • Dependency Failure: Simulating the failure of downstream APIs or databases.
  • Non-Gradual Failures: Instantaneous vs. slow degradation of a service.
• Exploring this spectrum builds resilience against the unexpected.

A validation technique where a physical hardware component (e.g., a robot's embedded motor controller) is integrated and tested within a simulated environment that provides realistic sensor inputs and receives actuator commands. This creates a hybrid testbed for stress-testing hardware logic under chaotic, simulated real-world conditions before full physical deployment.

• Core Purpose: To test the integration of real hardware with simulated dynamics and environments.
• Chaos Application: The simulated environment can be programmed to generate chaotic scenarios—sudden obstacle appearances, sensor noise spikes, or communication bus errors—to test the hardware's response.
• Benefit: Enables high-fidelity failure mode testing without the cost and risk of damaging a full physical robot.

A measure of how well the internal states of a system can be inferred from knowledge of its external outputs, achieved through the collection and analysis of logs, metrics, and traces. Chaos engineering is fundamentally dependent on high-fidelity observability; you cannot run a controlled experiment if you cannot see its effects.

• The Three Pillars:
  • Logs: Discrete events (e.g., 'E-stop triggered').
  • Metrics: Numerical measurements over time (e.g., CPU temperature, loop frequency).
  • Traces: End-to-end journey of a request through distributed services.
• For Robotics: Extends to physical telemetry like joint torque, battery voltage, and localization covariance, providing a unified view of software and hardware health during chaos experiments.

The part of a system's overall safety that depends on its components operating correctly in response to inputs, managed through rigorous design standards like ISO 26262 (automotive) or IEC 61508 (industrial). Chaos engineering serves as a practical, empirical validation method for FuSa assumptions and fault tolerance mechanisms.

• Safety Integrity Levels: Standards define Automotive Safety Integrity Levels (ASIL) or Safety Integrity Levels (SIL) that mandate specific probabilities of dangerous failure.
• Chaos as Validation: Chaos experiments can test safety mechanisms like watchdog timers, redundant controllers, and safe-state transitions under realistic fault conditions.
• Relationship: FuSa provides the prescriptive framework for designing for safety; chaos engineering provides the experimental methodology for verifying that safety in complex, emergent conditions.

A system's ability to produce the same output, within a bounded and predictable time frame, for a given set of inputs and initial conditions. This is a foundational requirement for reliable real-time control in robotics. Chaos engineering challenges this determinism by introducing unpredictable variables, testing the system's ability to either maintain control or degrade gracefully.

• Real-Time Requirement: Critical for control loops where missing a deadline can cause instability or failure.
• Chaos Impact: Experiments may inject non-deterministic elements (e.g., random CPU load, network jitter) to see if timing guarantees hold or if the system has adequate safeguards.
• Analysis: Involves measuring Worst-Case Execution Time (WCET) under chaotic conditions to verify schedulability.

A system engineering discipline focused on identifying anomalies, errors, or failures in a system and determining their root cause to enable recovery or safe shutdown. Chaos engineering actively generates potential faults to stress-test and improve these diagnostic systems.

• Detection vs. Diagnosis: Detection identifies that something is wrong; diagnosis determines what and why.
• Methods: Can range from simple threshold checks on sensor values to complex machine learning models analyzing system telemetry for anomalous patterns.
• Chaos Feedback Loop: By injecting known faults, engineers can verify that detection rules fire correctly, diagnostic logs are generated, and the system's operational status (e.g., 'degraded performance') is accurately reported.