Shielded Reinforcement Learning

A synchronous, deterministic component that sits between the agent's policy output and the environment. Before an action is executed, the shield performs a formal verification check against a set of predefined safety specifications. If the proposed action violates a constraint, the shield overrides it with a provably safe alternative. This creates a hard safety envelope that is completely independent of the learning process.

Safety constraints are defined using formal logic, typically Linear Temporal Logic (LTL) or Signal Temporal Logic (STL) . Unlike a reward function that merely penalizes unsafe behavior, these specifications are absolute rules. Examples include:

• G(velocity < 120) — Globally, velocity must always be less than 120.
• G(obstacle_distance > 0) — The agent must never collide. This makes safety auditable and provable, not just statistically likely.

When the shield blocks an unsafe action, it must provide a replacement. This is achieved through shield synthesis, which pre-computes a reactive system from the formal specification. The synthesized shield contains a winning strategy — a lookup table or logic that, for any state, outputs the set of all permissible actions. The override mechanism selects the safe action closest to the agent's original intent, minimizing interference.

Two temporal strategies exist for intervention:

• Preemptive Shielding: The shield evaluates the action before it is sent to the environment. This is the standard approach for preventing instantaneous catastrophic failures.
• Postposed Shielding: The shield allows the action but intervenes on the subsequent step if the state would become unrecoverable. This is useful when the immediate action is safe but leads inevitably to a failure state. The choice depends on the lookahead horizon of the specification.

The shield is not just a deployment guardrail; it actively shapes the learning process. During training, the agent observes the shielded action, not its own unsafe proposal. This means the agent's experience is always within the safe state space. Over time, the policy learns to internalize the shield's constraints, reducing the override frequency. This is known as shielded exploration, enabling safe learning in high-risk environments without needing a separate safe exploration policy.

The process of automatically generating a shield from a logical specification. Given an LTL formula φ, a synthesis algorithm constructs a deterministic finite automaton (DFA) that monitors the sequence of states and actions. The DFA transitions to a 'violation' state if φ is breached. The shield uses this automaton to filter actions in real-time. Tools like Slugs or custom SMT-solver-based synthesizers are used to compile the logic into executable code.

A formal shield acts as an unoverrideable safety layer between the RL policy and the vehicle's actuators. The shield monitors every proposed action—acceleration, steering angle, braking force—against a set of hard constraints derived from traffic laws and physics.

• Prevents red-light violations by overriding throttle commands when intersection states indicate a stop requirement
• Enforces minimum following distance by capping acceleration when LiDAR detects proximity below a safe threshold
• Guarantees collision-free lane changes by blocking steering actions that would intersect with occupied spatial zones

The shield operates at the control frequency (typically 100Hz), ensuring safety-critical interventions occur within microseconds. This architecture allows the RL agent to learn aggressive, efficient driving strategies while the shield provides a provable safety envelope.

In manufacturing cells where robots operate alongside humans, a shield enforces ISO 10218 safety standards by monitoring joint torque, velocity, and position commands against dynamic safety boundaries.

• Defines virtual safety cages as geometric constraints in the robot's configuration space
• Overrides velocity commands that would exceed power and force limiting thresholds when human proximity sensors are triggered
• Prevents self-collision by solving inverse kinematics through a constrained manifold that excludes joint configurations intersecting the robot's own structure

The shield synthesizes constraints from multiple sensor modalities—vision, torque sensing, and laser scanners—into a unified safety monitor. This enables high-speed operation during normal conditions with guaranteed safe degradation when humans enter the shared workspace.

Shielded RL agents control generator setpoints and battery storage dispatch to maintain grid frequency at 50/60 Hz. The shield encodes N-1 reliability criteria as hard constraints on the action space.

• Prevents cascading failures by blocking dispatch commands that would overload any single transmission line beyond its thermal rating
• Enforces voltage stability margins by constraining reactive power commands to keep bus voltages within ±5% of nominal
• Guarantees minimum spinning reserve by overriding generator shutdown commands when reserve capacity would drop below regulatory requirements

The shield uses a linearized optimal power flow model to verify constraint satisfaction in real-time, providing mathematically provable safety guarantees while the RL agent optimizes for economic efficiency and renewable integration.

An RL agent learns personalized ventilation strategies for ICU patients, while a shield enforces clinical safety guidelines derived from established medical protocols.

• Blocks tidal volume adjustments that would cause barotrauma by exceeding patient-specific lung compliance limits
• Prevents oxygen toxicity by constraining FiO₂ settings below 60% for extended durations
• Enforces minimum minute ventilation to prevent respiratory acidosis based on continuous blood gas monitoring

The shield's constraints are parameterized by patient-specific physiology—weight, lung compliance, and arterial blood gas measurements—creating a personalized safety envelope. This allows the RL agent to explore aggressive weaning strategies that reduce ventilation time while the shield provides clinician-auditable safety guarantees.

Multiple RL agents control individual drones in a coordinated swarm, with each agent's shield enforcing collision avoidance and airspace boundary constraints.

• Uses control barrier functions to guarantee pairwise separation distances never drop below a minimum threshold, even during aggressive formation changes
• Enforces geofencing by projecting proposed waypoints onto the convex hull of the authorized airspace when boundary violations are predicted
• Prevents downwash interference by constraining vertical separation between drones in stacked formations

The shield operates in a decentralized manner on each drone's flight controller, ensuring safety even during communication dropouts. This architecture enables dense swarm configurations for light shows, surveillance, and package delivery with formal safety proofs.

An RL agent learns order execution strategies to minimize market impact, while a shield enforces MiFID II and SEC regulatory constraints on every order submission.

• Prevents wash trading by blocking buy and sell orders for the same instrument within prohibited time windows
• Enforces position limits by capping order quantities that would cause aggregate exposure to exceed exchange-mandated thresholds
• Guarantees minimum quote life by overriding cancellation commands that would violate order resting time requirements

The shield operates as a pre-trade risk check with deterministic latency, ensuring compliance verification completes before any order reaches the exchange. This provides an auditable compliance layer that regulators can inspect independently from the proprietary trading strategy.