Recovery Policy

Recovery policies can be reactive, triggered after a failure or constraint violation is detected, or proactive, which anticipate and avoid unsafe states.

• Reactive: Executes a pre-defined recovery maneuver (e.g., a controlled stop, a reset sequence) upon fault detection.
• Proactive: Uses a safety critic or predictive model to steer the system away from the boundary of the safe set before a violation occurs, often integrated with Control Barrier Functions (CBFs).

High-assurance recovery policies are often designed with formal methods to provide mathematical guarantees of safety.

• Lyapunov functions and Barrier functions are used to prove stability and enforce set invariance.
• Formal verification techniques may be applied to prove that the policy will always bring the system back to a safe region from any state within a defined recovery domain.

In Safe RL, a recovery policy is a key component for handling constraint violations defined in a Constrained Markov Decision Process (CMDP).

• It acts as a fallback policy when the primary learning agent's action would violate a safety cost constraint.
• This enables training in high-risk simulation environments by providing a safe 'reset', crucial for Sim-to-Real Transfer Learning where exploring failure states is necessary but must be contained.

Effective recovery often requires switching between different control regimes or abstraction levels.

• A hierarchy may involve low-level joint control for stabilization, mid-level gait correction, and high-level task re-planning.
• The policy must select the appropriate recovery mode (e.g., 'brace for impact', 'soft reset', 're-grasp') based on the diagnosed failure type and context.

Recovery policies are extensively trained and validated within physics simulation engines using fault injection and adversarial robustness testing.

• Domain randomization is applied to the failure conditions to ensure the policy generalizes to unseen real-world faults.
• Sim-to-Real Benchmarking metrics evaluate how well the simulated recovery performance transfers to physical hardware, a core part of Digital Twin Creation.

Deployment relies on continuous runtime monitoring to detect conditions requiring recovery.

• Monitors check for Out-of-Distribution (OOD) states, constraint violations, or system fault flags.
• This enables shielded learning architectures, where a monitor overrides unsafe actions with the recovery policy, ensuring graceful degradation into a fail-safe mode rather than a catastrophic failure.

The subfield of reinforcement learning focused on developing algorithms that learn to maximize performance while satisfying explicit safety constraints. Recovery policies are often a key component or output of Safe RL algorithms.

• Core Problem: Formalized using Constrained Markov Decision Processes (CMDPs), where the objective is to maximize reward subject to cost constraints.
• Methods: Include constrained policy optimization, Lagrangian methods, and risk-sensitive formulations.
• Relation to Recovery: A recovery policy can be the learned strategy for a CMDP where the primary cost is associated with leaving a safe region of the state space.

A mathematical construct used in control theory to formally guarantee that a dynamical system's state remains within a predefined safe set. It synthesizes a minimally invasive safe control input.

• Mechanism: A CBF defines a safe set. The controller is filtered to ensure the derivative of the CBF keeps the system within this set.
• Formal Guarantee: Provides a certificate of forward invariance for the safe set.
• Relation to Recovery: A recovery policy can be implemented using a CBF framework. When the system nears an unsafe state, the CBF-based controller overrides the nominal policy to 'push' the state back toward safety, acting as an automated recovery mechanism.

A safety technique involving the continuous, real-time observation of a system's execution to detect violations of specified safety properties or constraints.

• Function: Acts as a watchdog, checking system state against a formal specification (e.g., "joint torque shall not exceed X Nm").
• Output: A binary signal indicating a safety violation or a quantitative measure of risk.
• Relation to Recovery: The runtime monitor is the trigger for a recovery policy. Upon detecting a constraint violation or predicting an imminent one, it signals the system to switch from its primary policy to the dedicated recovery policy.

An approach to safe reinforcement learning where a 'shield'—a pre-computed or learned runtime monitor—intervenes to override potentially unsafe actions proposed by the learning agent.

• Architecture: The shield sits between the agent's policy and the environment. It can correct, modify, or replace unsafe actions.
• Types: Shields can be hard (absolute prevention) or soft (probabilistic correction).
• Relation to Recovery: A shield is a form of recovery policy. It is a reactive policy that maps unsafe proposed actions to the nearest safe action, effectively recovering from a potentially hazardous decision before it is executed.

A predefined, minimal-risk condition or state that a system is designed to enter upon the detection of a fault, failure, or irrecoverable error.

• Design Principle: Part of fault-tolerant system design. Examples include a robot locking its brakes, a drone initiating an emergency landing, or a controller switching to a conservative, verified backup.
• Objective: To prevent harm when continued normal operation is impossible or dangerous.
• Relation to Recovery: A fail-safe mode is the terminal state of a recovery policy. If the recovery policy cannot successfully guide the system back to a normal safe region, its final action is to transition the system into its fail-safe mode.

The fundamental mathematical framework used to formalize Safe Reinforcement Learning problems. It extends the standard MDP by adding constraints on expected cumulative costs.

• Formalism: Defined by the tuple (S, A, P, R, C, d), where C is a cost function and d is a constraint threshold.
• Objective: Find a policy π that maximizes expected cumulative reward subject to the expectation of cumulative cost being less than d.
• Relation to Recovery: A recovery policy can be explicitly learned as the optimal policy for a CMDP where:
  • The state space includes error and unsafe regions.
  • The reward is for returning to a safe set.
  • The cost may penalize time spent unsafe or dangerous actions.