Safety Constraints

Constrained RL formulates the learning problem with explicit safety limits. Instead of just maximizing cumulative reward, the policy must also satisfy constraints on expected costs (e.g., joint torque limits, proximity to obstacles).

• Key Algorithms: Constrained Policy Optimization (CPO), Lagrangian methods which add constraint violations as penalty terms to the objective.
• Implementation: Constraints are defined as cost functions. The optimization becomes: maximize reward subject to keeping expected costs below a threshold.
• Example: A robotic arm policy trained to reach a target is constrained to keep its calculated collision probability below 0.01 per timestep.

CBFs are mathematical functions used to formally guarantee a system stays within a safe set. They act as a runtime filter on any proposed action.

• Mechanism: A CBF, h(x), defines a safe state where h(x) ≥ 0. For any commanded action, a quadratic program (QP) finds the nearest safe action that ensures the time derivative of h(x) keeps the state safe (ḣ(x) ≥ -α(h(x))).
• Real-time Enforcement: This safety filter runs at the control frequency (e.g., 100Hz), correcting potentially unsafe actions from a learned policy.
• Use Case: Ensuring a quadruped robot's foot never swings below ground level, regardless of the policy's output.

MPC is an online optimization-based controller that explicitly enforces constraints over a finite time horizon.

• How it works: At each control step, MPC solves an optimization problem to find a sequence of actions that minimizes a cost (e.g., deviation from a path) while respecting system dynamics and hard constraints (e.g., velocity limits, obstacle boundaries). Only the first action is executed.
• Integration with Learned Policies: A learned policy can provide a reference trajectory or cost function for the MPC to track, while MPC handles the precise constraint satisfaction.
• Advantage: Provides provable constraint satisfaction for linear or convex models, and can react to newly perceived obstacles.

These are simple, effective runtime techniques to prevent physically impossible or dangerous commands from being sent to actuators.

• Action Masking: The policy outputs actions in a permissible space. Before execution, a mask forcibly sets unsafe action dimensions to zero or a safe value (e.g., masking gripper force if no object is detected).
• Action Clipping: The continuous output of a policy is clamped between pre-defined minimum and maximum values (e.g., action = np.clip(policy_output, -max_torque, max_torque)).
• Application: Universal first-layer safety for any learned policy. It prevents hardware damage from outlier commands but does not reason about complex state-dependent constraints.

A layered architecture where a dedicated, verifiably safe policy takes over when the primary policy is likely to violate constraints.

• Architecture: A monitor (e.g., an uncertainty estimator or constraint checker) observes the primary policy's state and intended action. If a safety violation is predicted, it triggers a switch to a recovery policy.
• Recovery Policy: Typically a simple, robust controller with a guaranteed safe behavior, such as braking to a stop, moving to a home position, or entering a torque-free gravity compensation mode.
• Design Principle: This follows the simplex architecture pattern, ensuring a safe baseline exists independently of the complex, high-performance primary policy.

This method modulates a policy's aggressiveness or triggers caution based on its estimated uncertainty, preventing action in unfamiliar, potentially unsafe states.

• Uncertainty Sources: Epistemic (model uncertainty) and Aleatoric (sensor noise). High epistemic uncertainty often indicates the robot is in a state not well-covered by training data.
• Implementation:
  • Use an ensemble of policies or a Bayesian Neural Network to estimate prediction variance.
  • Scale down action magnitudes proportionally to uncertainty.
  • Trigger a switch to a safe, conservative mode if uncertainty exceeds a threshold.
• Benefit: Creates a natural 'safety buffer' in states far from the training distribution of the simulation.

The reality gap (or sim2real gap) is the performance discrepancy between a policy's behavior in simulation and its behavior on physical hardware. It is caused by inaccuracies in the simulated model, including simulation bias in physics, rendering, or sensor models. This gap necessitates the use of transfer techniques like domain randomization and safety constraints to ensure the policy remains effective and safe when the world differs from its training environment.

Domain randomization is a core sim-to-real technique where a policy is trained in a simulation with randomized environmental parameters (e.g., textures, lighting, friction coefficients, masses). This forces the policy to learn robust features that are invariant to these visual and dynamic variations, rather than overfitting to simulation artifacts. It is a proactive method for bridging the reality gap and is often used in conjunction with safety constraints to prepare policies for the unpredictability of the real world.

Hardware-in-the-Loop (HIL) testing is a critical validation step before full deployment. Physical robot hardware (actuators, sensors) is connected in real-time to a simulation that provides virtual environmental feedback. This allows for:

• Safe stress-testing of policies against edge cases and failure modes.
• Validation of low-level control latency and hardware interfaces.
• Collection of real-world data for system identification or offline adaptation, all within a controlled, repeatable framework that enforces safety constraints before physical interaction.

Uncertainty quantification involves estimating the confidence or error bounds of a model's predictions. In sim-to-real transfer, it is essential for safe operation. Techniques include:

• Bayesian neural networks that provide predictive variance.
• Ensemble methods where disagreement among multiple policies indicates uncertainty.
• Out-of-distribution detection to flag when the robot encounters states far from its training data. High uncertainty can trigger safety constraints, defaulting to a conservative fallback policy or requesting human intervention.

Model Predictive Control (MPC) is an advanced online control method that uses an explicit (often simplified) model of the system to predict future states over a finite horizon. At each control cycle, it solves an optimization problem to find the optimal sequence of actions, typically subject to explicit safety constraints (e.g., joint limits, obstacle avoidance). The first action is executed, and the process repeats. MPC is highly effective for managing dynamic constraints and can be used as a safety wrapper for a learned neural network policy.

Shadow mode deployment is a low-risk rollout strategy for validating policies in the real world. The new policy runs in parallel with the existing, proven production system (e.g., a classical controller). It receives the same real-world sensor inputs and generates proposed actions, but these actions are not executed on the physical robot. Instead, its decisions are logged and compared against the live system. This allows engineers to:

• Validate performance and safety constraint adherence in real conditions.
• Identify edge cases and covariate shift without operational risk.