Execution Monitoring

The State Estimator fuses raw sensor data (e.g., from LiDAR, cameras, encoders, IMUs) to produce a consistent, real-time estimate of the robot's internal state and the state of key objects in its environment. It resolves sensor noise and uncertainty, often using probabilistic filters like a Kalman Filter or Particle Filter, to provide the single source of truth for the robot's pose, velocity, and object positions. This estimated state is the primary input for all subsequent monitoring logic.

This component forecasts the expected future state of the system based on the current plan. It uses the robot's dynamic model and the planned trajectory to simulate forward in time, generating a series of expected sensor readings, joint positions, or end-effector poses. The monitor continuously compares these predictions against the actual state reported by the State Estimator. Significant divergence indicates a potential execution failure, such as an unexpected obstacle or actuator slippage.

The Condition Monitor evaluates a set of logical and temporal predicates against the current and historical state. These predicates define the execution invariants and safety constraints that must hold true for the plan to remain valid. Examples include:

• Geometric Constraints: "End-effector must remain >10cm from the table surface."
• Temporal Constraints: "The grasp action must complete within 2 seconds."
• Logical Preconditions: "Object A must be securely held before moving to location B." Violation of any monitored condition triggers a fault.

When a discrepancy is detected, this component analyzes its pattern to classify the fault type and diagnose the probable root cause. It moves beyond simple anomaly detection to reasoning. For instance, it distinguishes between:

• Perceptual Faults: A sensor failure or persistent occlusion.
• Actuation Faults: A motor stall or gripper slip.
• Environmental Faults: An object being moved by an external agent.
• Model Inaccuracy: A discrepancy between the robot's dynamic model and its real-world behavior. Accurate diagnosis is critical for selecting the appropriate recovery strategy.

Based on the diagnosed fault and the current context, this component selects and parameterizes a recovery policy from a predefined library. The selection is a decision-making process that considers the severity of the fault, remaining task objectives, and available resources. Common recovery policies include:

• Replanning: Requesting a new plan from the task planner from the current state.
• Retrying: Re-executing the failed action, potentially with adjusted parameters.
• Escalation: Pausing execution and requesting human operator intervention.
• Graceful Degradation: Switching to a simplified, safer behavior mode.

This is the observability backbone. It continuously streams timestamped telemetry data—including raw sensor readings, estimated states, predicted states, condition evaluations, and fault events—to persistent storage and dashboards. This data is essential for:

• Post-Mortem Analysis: Debugging failures and improving models.
• Performance Benchmarking: Measuring plan execution accuracy and monitor latency.
• Model Improvement: Providing labeled failure data to refine the robot's dynamic or perceptual models.
• Compliance & Audit: Providing a verifiable record of the system's decision-making process during operation.

In automated fulfillment centers, execution monitoring systems track the planned path of Autonomous Mobile Robots (AMRs) against real-time LiDAR and camera feeds. Deviation detection algorithms identify unexpected obstacles—like a fallen pallet or a human worker—triggering an immediate replanning request. This real-time feedback loop prevents costly collisions and ensures continuous operation, with systems often achieving sub-100 millisecond reaction times to initiate evasive maneuvers.

For robotic arms performing delicate assembly tasks (e.g., inserting a circuit board), execution monitoring compares expected end-effector pose and force-torque sensor readings against live data. Temporal logic monitors check if sub-tasks complete within specified time windows. A deviation, such as excessive resistance indicating a misaligned part, triggers a predefined recovery behavior (e.g., a gentle retry) or halts the line, preventing part damage. This is a key component of error detection and recovery (EDR) systems in manufacturing.

In self-driving cars, execution monitoring acts as a continuous safety supervisor. It validates that the vehicle's perceived state (from cameras, radar) aligns with the planned trajectory and that all safety envelopes (e.g., minimum following distance, maximum curvature) are respected. If the system detects a perception-planning mismatch—such as an object appearing where none was predicted—or a violation of a safety rule, it can initiate a minimal risk condition (MRC), like a controlled stop. This layer is critical for functional safety standards like ISO 26262.

In robot-assisted surgery, execution monitoring ensures the robotic system adheres strictly to the surgeon's pre-operative plan. It tracks instrument tip positions in real-time against a 3D model of the patient's anatomy. Geofencing virtual fixtures create software-defined boundaries that prevent the tool from moving into critical, no-fly zones. Any attempt to breach these boundaries triggers haptic feedback to the surgeon and can automatically dampen or stop motor commands, providing a vital safety interlock during high-precision operations.

For rovers on Mars, communication delays make real-time human intervention impossible. Execution monitoring is the rover's primary safeguard. It continuously checks system health telemetry (power, temperature), verifies that observed terrain matches expectations from local maps, and confirms that each commanded action (e.g., a wheel turn) produces the expected odometry change. Upon detecting a fault—like a wheel digging in unexpectedly—the monitor triggers a fault protection response, such as stopping all motion and entering a safe state to await new instructions from Earth.

In a coordinated fleet of drones or ground robots, execution monitoring occurs at both the individual agent and fleet level. Each robot monitors its own task progress, while a central or distributed supervisor monitors for inter-agent conflicts (e.g., two robots heading to the same resource) and global constraint violations (e.g., exceeding a total power budget). Detected conflicts trigger re-allocation or negotiation protocols among agents. This ensures the overall mission's temporal and spatial constraints are maintained despite individual uncertainties.

The verification process that occurs before execution to ensure a generated plan is logically sound, feasible, and safe. It often uses simulation or formal methods to check for:

• Logical consistency: The sequence of actions achieves the goal.
• Kinematic feasibility: The robot can physically perform the motions.
• Collision-free paths: The planned trajectory avoids all known obstacles.
• Temporal constraints: All timing and concurrency requirements are satisfied.

Plan validation is a pre-execution safeguard, while execution monitoring is the real-time safeguard during the plan's physical enactment.

The process of generating a new plan from the robot's current state when the original plan fails or the environment changes unexpectedly. It is triggered by execution monitoring systems that detect:

• Persistent plan failures: An action cannot be completed after several retries.
• Unmodeled obstacles: New objects block the planned path.
• Goal condition changes: The target state is modified during operation.
• Resource exhaustion: A required tool breaks or a battery depletes.

Effective replanning requires a tight feedback loop with the execution monitor, which provides the failure context and the current world state to the planner.

The set of all possible states that a dynamical system, such as a robot, can occupy. It is defined by a vector of state variables, which for a mobile manipulator might include:

• Configuration variables: Joint angles, wheel positions.
• Dynamic variables: Joint velocities, acceleration.
• Environmental variables: Position of perceived objects, door states.

Execution monitoring fundamentally involves observing the robot's trajectory through this state space and comparing it to the expected trajectory defined by the plan. Deviations indicate potential failures.

An advanced control method where a dynamic model of the system is used to predict future behavior over a short time horizon. At each control cycle, it solves an optimization problem to determine the optimal immediate control inputs. MPC is deeply related to execution monitoring because:

• It continuously re-plans at a low level, inherently monitoring for deviations from its own predicted model.
• It can incorporate constraints (like obstacle avoidance) that act as continuous monitors.
• The prediction horizon provides a short-term expectation against which sensor feedback is compared, a core monitoring function.

While MPC handles low-level trajectory tracking, execution monitoring often operates at a higher, more discrete action level.

A directed graph representation of a plan where nodes represent tasks or actions and edges represent dependencies (temporal, causal, or resource-based). This structure is critical for execution monitoring as it provides the expected causal and temporal framework for observation.

Monitors use the task graph to:

• Check precondition satisfaction before an action node is executed.
• Verify postcondition achievement after an action node completes.
• Understand dependency violations if a downstream action fails because an upstream one did not produce its expected effects.
• Manage concurrent execution of nodes, monitoring for resource conflicts.

The real-time algorithmic process of dynamically adjusting a robot's planned trajectory to prevent contact with unexpected or moving obstacles. It is a specialized, low-latency form of execution monitoring and reaction focused on a single critical failure mode: imminent collision.

Key techniques include:

• Potential fields: Creating repulsive forces around obstacles.
• Velocity obstacles: Calculating velocities that would lead to a collision.
• Dynamic window approach: Evaluating feasible velocities within a short time window.

Collision avoidance typically runs as a high-priority, reactive layer beneath a higher-level execution monitor, which may handle more complex, semantic failures (e.g., 'failed to insert peg' vs. 'about to hit wall').