Root Cause Analysis (RCA)

A foundational iterative questioning technique used to drill down through layers of symptoms to uncover a root cause. Each answer forms the basis of the next 'why?' question.

• Example for a robot failure:
  1. Why did the AMR stop? Its motor controller overheated.
  2. Why did it overheat? The cooling fan was not spinning.
  3. Why wasn't the fan spinning? The bearing was seized due to dust ingress.
  4. Why was there dust ingress? The intake filter was missing.
  5. Why was the filter missing? The maintenance checklist was not followed during the last service.
• Best for: Simple to moderately complex issues where cause-and-effect relationships are relatively linear.

A visual cause-and-effect diagram that categorizes potential causes of a problem to stimulate systematic brainstorming. The 'head' of the fish is the problem statement, with 'bones' representing major cause categories.

• Standard Categories (6Ms):
  • Method: Process or procedural flaws.
  • Machine: Hardware or equipment failure.
  • Material: Issues with consumables or parts.
  • Manpower/People: Human error or training gaps.
  • Measurement: Inaccurate sensors or metrics.
  • Environment/Milieu: External factors like temperature or network congestion.
• Fleet Application: Ideal for analyzing complex fleet incidents where multiple system components (agents, network, orchestration software, environment) could be contributing factors.

A top-down, deductive failure analysis method that uses Boolean logic to model the pathways within a system that can lead to a specific, undesired top-level event (e.g., 'Fleet-Wide Deadlock').

• Key Components:
  • Top Event: The system-level failure being analyzed.
  • Basic Events: The fundamental, indivisible failures (e.g., 'Agent 07 battery depleted', 'Wi-Fi AP 3 offline').
  • Logic Gates (AND/OR): Show how combinations of lower-level events lead to higher-level events.
• Quantitative Use: Can incorporate failure rate data to calculate the probability of the top event. This is critical for Safety Integrity Level (SIL) assessments in physical systems.
• Best for: High-consequence failures, safety-critical systems, and understanding complex failure cascades in multi-agent orchestration.

A technique focused on identifying what changed in the system before an incident occurred. The core premise is that changes often introduce failures.

• Process:
  1. Document the exact state of the system when it was working correctly (the 'known good' baseline).
  2. Document the state when the failure occurred.
  3. Systematically compare the two states to identify all differences.
• Relevant Changes in a Fleet Context:
  • Software/Firmware: New OTA updates to agents or the orchestration platform.
  • Configuration: Changes to path planning parameters, zone definitions, or agent priorities.
  • Infrastructure: Network topology changes, database schema migrations.
  • Physical Environment: New obstacles, modified warehouse layouts.
  • Operational Load: A significant spike in task volume or complexity.
• Best for: Incidents that occur suddenly in a previously stable environment.

Examines the controls or safeguards (barriers) that were in place to prevent a failure and determines why they were ineffective or absent. It shifts focus from 'what failed' to 'what failed to prevent the failure.'

• Types of Barriers:
  • Physical: Guardrails, hardware interlocks.
  • Administrative: Procedures, checklists, training.
  • Software: Input validation, circuit breaker patterns, liveness probes.
  • Natural: Physical separation of agents.
• Analysis Questions:
  • What barriers should have been in place?
  • Which barriers were in place?
  • Why did each barrier fail? (Not built, not used, not effective, overridden)
• Fleet Example: A collision occurred. Barriers might have included: zone management software (failed due to bug), proximity sensors (failed due to dirt), and speed limits (overridden by a priority task). Analysis reveals the layered defense was compromised.

A structured, non-linear method based on the principle that every cause is an action. It focuses on describing the cause-and-effect relationships in a precise language to build a visual causal graph.

• Core Components:
  • Cause: Always expressed as a Subject-Action-Object triplet (e.g., 'Dust - blocks - air filter', 'Engineer - skips - maintenance step').
  • Effect: The result of that action.
  • Causal Relationships: Effects become causes for subsequent effects, creating a chain or web.
• Key Differentiator: It explicitly seeks to identify all causes, not just one 'root', recognizing that problems often have multiple contributing causal chains. This is highly applicable to complex, software-driven systems like heterogeneous fleets where failures are rarely monocausal.
• Outcome: Produces a detailed causal factor chart that maps the entire incident ecosystem, ideal for systemic process improvement.

The process of identifying patterns in agent telemetry that deviate significantly from established baselines, serving as the primary trigger for initiating a Root Cause Analysis. Anomaly detection algorithms (statistical, machine learning-based) analyze metrics like latency, error rates, battery drain, and positional drift to flag potential faults before they cascade into system-wide failures. In fleet operations, this might detect an autonomous mobile robot (AMR) with abnormally high motor temperature or a manual forklift transmitting erratic GPS data.

The continuous, real-time flow of operational data from all agents to a central collection system. This data is the foundational evidence for RCA. A comprehensive stream includes:

• Metrics: Numerical data (e.g., State of Charge, CPU load, wheel encoder counts).
• Logs: Structured event records from agent software.
• Traces: Distributed request paths for multi-agent tasks.
• Events: Discrete occurrences like 'task accepted' or 'obstacle detected'. Without a high-fidelity, low-latency telemetry stream, RCA lacks the granular temporal data needed to reconstruct failure sequences.

A monitoring method that follows a single logical operation (e.g., 'retrieve item from bin A17') as it propagates through multiple services and agents. For RCA, traces provide causal links. If a picking task fails, a trace shows the exact path: from the orchestration middleware issuing the command, to the AMR accepting it, to its navigation stack planning a route, and finally to its gripper service timing out. This end-to-end visibility is essential for distinguishing between a root cause (e.g., a network partition at the warehouse rack) and a downstream symptom (the gripper command never arrived).

A proactive strategy that uses data analysis and machine learning to forecast equipment failures, directly informing RCA by shifting focus from 'what broke' to 'what is degrading.' Predictive maintenance models analyze historical telemetry (vibration, temperature, power cycles) to estimate Remaining Useful Life (RUL) for components like LiDAR sensors or drive motors. An RCA initiated by a predicted failure has a clearer investigative path—the analysis centers on validating the model's prediction and confirming the specific degradation mode (e.g., bearing wear vs. lubrication loss).

A stability design pattern that prevents a failing service from causing cascading system failures, creating clean failure boundaries that simplify RCA. When an agent's API calls to a central map service repeatedly time out, the circuit breaker trips and fails fast, preventing the agent from hanging. For an engineer performing RCA, this pattern localizes the fault: the agent's 'failure to navigate' is correctly identified as a symptom, and the root cause investigation focuses upstream on the map service or the network link to it, rather than on the agent's local software.

A foundational, iterative interrogative technique used to drill down to the root cause of a problem by repeatedly asking 'Why?' It provides structure to the RCA process. In a fleet context:

1. Why did AMR-23 stop? Its safety laser triggered an emergency stop.
2. Why did the laser trigger? It detected an obstacle in its path.
3. Why was an obstacle there? A manual pallet jack was parked in a transit lane.
4. Why was it parked there? The driver was responding to a separate task alert.
5. Why was that task routed there? The dynamic task allocation system had stale zone occupancy data. The root cause is often a systemic or procedural issue (data staleness), not the immediate technical fault (laser stop).