Root Cause Analysis (RCA)

A foundational iterative questioning technique used to drill down through layers of symptoms to uncover a root cause. It is most effective for simple, linear problems.

• Process: Start with the problem statement and ask "Why did this happen?" repeatedly (typically five times) until the underlying process or decision failure is revealed.
• Example: A robot fails to dock. Why? Its battery was depleted. Why? It missed its scheduled charging slot. Why? The charging station was occupied by another agent. Why? The fleet scheduler did not account for charging contention. Why? The scheduling algorithm lacks a battery-aware constraint model (Root Cause).

A visual cause-and-effect analysis tool that organizes potential causes into categorical branches, facilitating brainstorming in complex scenarios. It is ideal for multidisciplinary team investigations.

• Standard Categories: Often use the 6 Ms: Method (process), Machine (hardware/software), Material (data/inputs), Manpower (human factors), Measurement (metrics/sensors), and Mother Nature (environment).
• Application: For a fleet-wide localization failure, branches might include: Sensor calibration (Machine), Map data staleness (Material), Communication latency (Method), and RF interference from new equipment (Mother Nature).

A top-down, deductive failure analysis method that uses Boolean logic to model the pathways to a specified system-level failure (the top event). It quantifies failure probabilities and identifies single points of failure.

• Logic Gates: Uses AND and OR gates to combine lower-level events (e.g., component failures, human errors).
• Use Case: Critical for safety-critical systems. To analyze a "collision between AMR and manual forklift," the tree would decompose into events like: Perception system fault OR (Path planning error AND Zone management failure) OR Communication link timeout.

A proactive, systematic method for evaluating a system before failures occur to identify where and how it might fail, and to prioritize improvements based on risk.

• Risk Priority Number (RPN): Calculated as Severity x Occurrence x Detection. Teams address high-RPN failure modes first.
• Proactive Application: Applied during the design of an orchestration middleware to score potential failures like: "Task allocation deadlock" (High Severity, Medium Occurrence, Low Detection = High RPN) leading to the design of a dedicated deadlock detection service.

A sequential, timeline-based method that maps the chain of events and conditions leading to an incident. It distinguishes between causal factors (necessary contributors) and the root cause.

• Process: Investigators build a detailed flowchart from the normal sequence of events, inserting deviations that led to the failure. Conditional statements (e.g., 'IF the battery API was unresponsive') are added alongside actions.
• Benefit: Provides an unambiguous narrative for complex, multi-step incidents in distributed systems, such as a cascading failure across a multi-agent fleet following a network partition.

A technique focused on identifying the defenses or controls that should have prevented an incident but failed. It is rooted in safety engineering models like James Reason's Swiss Cheese Model.

• Key Questions: What barriers were in place (e.g., input validation, hardware interlocks, sanity checks)? How did each barrier fail? Why was the barrier inadequate or bypassed?
• Example: An AMR executes an incorrect payload drop. Barriers may include: 1) Task verification API (failed: returned stale data), 2) Load sensor check (failed: sensor fault), 3) Geofence software limit (bypassed: bug in coordinate transformation). The analysis reveals both technical and procedural root causes.

A software design pattern that detects failures and prevents an application from repeatedly attempting an operation that is likely to fail. It acts as a proxy for operations that can fail, opening the circuit after failures exceed a threshold to allow the failing subsystem time to recover.

• Key Mechanism: Toggles between Closed, Open, and Half-Open states based on failure counts.
• Purpose: Prevents cascading failures and resource exhaustion by failing fast.
• Example: An API client stops calling a downed payment service after 5 consecutive timeouts, periodically probing for recovery.

A Retry Policy defines the conditions (which errors, how many times) under which a failed operation is reattempted. Exponential Backoff is a specific algorithm for calculating wait times between retries, increasing the delay exponentially (e.g., 1s, 2s, 4s, 8s).

• Purpose: To handle transient faults (network glitches, temporary load) gracefully.
• Critical Consideration: Must be combined with idempotency to avoid duplicate side effects.
• Jitter: Often added to backoff calculations to prevent retry storms from synchronized clients.

A persistent queue that stores messages or tasks that cannot be processed successfully after exhausting all retry attempts. It acts as a quarantine zone for problematic items, enabling offline RCA without blocking the main processing flow.

• Primary Use: Capturing poison pills or messages causing persistent errors for manual or automated analysis.
• Integration: Essential in event-driven and message-based architectures (e.g., Apache Kafka, Amazon SQS).
• RCA Link: The DLQ is a primary data source for investigating root causes of processing failures.

A design pattern for managing data consistency across multiple services in a distributed transaction. Instead of a two-phase commit, it uses a sequence of local transactions, each with a compensating transaction (rollback action) that is triggered if a subsequent step fails.

• Choreography: Services produce events that trigger the next step.
• Orchestration: A central coordinator manages the sequence.
• RCA Relevance: A saga failure requires analyzing which compensating transaction failed and why, which is a complex RCA scenario in microservices.

A resilience pattern inspired by ship compartments that isolate elements of an application into pools. Failure or resource exhaustion in one pool (bulkhead) is contained and does not drain resources from or cascade to others.

• Implementation: Using separate thread pools, connection pools, or even microservice instances for different workloads or clients.
• Benefit: Prevents a single failing component from taking down the entire system.
• RCA Impact: Limits the blast radius of an incident, making the faulty subsystem easier to isolate and analyze.

A key Site Reliability Engineering (SRE) metric measuring the average time taken to restore a service to normal operation after a failure. It encompasses detection, diagnosis, mitigation, and restoration.

• Components: Includes Mean Time To Detect (MTTD) and Mean Time To Repair (MTTR).
• RCA's Role: Effective RCA directly reduces MTTR by speeding up the diagnosis phase and leading to permanent fixes that prevent recurrence.
• Business Impact: Lower MTTR is directly correlated with higher service availability and customer satisfaction.