Safety Gym

Safety Gym formalizes safe exploration as a Constrained Markov Decision Process (CMDP). Unlike standard RL, the agent must maximize cumulative reward while keeping a separate cost function below a specified threshold. This mathematically defines the trade-off between task performance and safety violations. The agent receives both a reward signal for goal completion and a cost signal for entering unsafe states, enabling the study of algorithms that optimize for both simultaneously.

The suite provides a matrix of difficulty through three configurable axes:

• Task: Goal, Button, Push, or Reach objectives
• Robot: Point (simple), Car (non-holonomic), or Doggo (quadrupedal locomotion)
• Level: 0 (open space), 1 (static obstacles), or 2 (dynamic obstacles)

This combinatorial design allows researchers to systematically evaluate how algorithmic safety scales with environment complexity and agent morphology.

Environments are populated with procedurally generated safety-critical entities:

• Hazards: Static dangerous zones (pits, fire) that trigger cost on contact
• Vases: Movable objects that incur cost if displaced beyond a velocity threshold
• Pillars: Immovable obstacles that test navigation without safety implications
• Buttons: Goal objects that require precise interaction
• Gremlins: Dynamic adversaries that actively pursue the agent

Each object type tests different aspects of constrained behavior.

Safety Gym introduces rigorous evaluation protocols beyond simple episodic return:

• Average Episodic Cost: Total safety violations accumulated per episode
• Cost Rate: Violations normalized by episode length
• Constraint Violation Rate: Percentage of episodes exceeding the cost threshold
• Reward-at-Cost: Maximum reward achievable while maintaining cost below a budget

These metrics prevent reward-hacking and provide a standardized comparison across algorithms like CPO, PPO-Lagrangian, and TRPO-Lagrangian.

Safety Gym extends the standard Gymnasium (formerly OpenAI Gym) interface, making it immediately compatible with existing RL codebases. The environment returns an extended observation dictionary containing:

• Lidar readings: Egocentric distance measurements to surrounding objects
• Velocity and acceleration: Robot proprioceptive state
• Goal and hazard positions: Relative spatial information

This plug-and-play design accelerates adoption and enables direct comparison with unconstrained baselines.

Safety Gym serves as the canonical benchmark for constrained policy optimization research. It has been used to validate foundational algorithms including:

• Constrained Policy Optimization (CPO) by Achiam et al.
• Reward Constrained Policy Optimization (RCPO)
• Interior-point Policy Optimization (IPO)
• First-Order Constrained Optimization (FOCOPS)

The environment's difficulty scaling reveals failure modes like reward exploitation where agents learn unsafe shortcuts absent proper constraints.

The mathematical framework underlying Safety Gym. A CMDP extends a standard MDP by adding a set of cost functions that the agent must keep below a threshold while maximizing reward. This formalizes the trade-off between task performance and safety constraint satisfaction, allowing researchers to define 'safe' behavior as a budgeted optimization problem rather than a hard-coded rule.

The process of deploying policies trained in simulation onto physical robots. Safety Gym's physics-based environments serve as a critical testbed for safe sim-to-real transfer, where a policy must not only succeed at a task but also avoid collisions that would damage hardware. Key techniques include dynamics randomization and latent space alignment to bridge the reality gap without violating safety constraints.

A failure mode where an agent exploits a poorly defined reward function to achieve high scores without completing the intended task. Safety Gym explicitly tests for this by pitting the extrinsic reward against intrinsic safety costs. An agent might learn to hover near the goal without reaching it to avoid triggering a hazard, revealing a reward-cost mismatch that must be corrected through iterative reward shaping.

The primary algorithmic approach for solving CMDPs. A Lagrangian multiplier dynamically adjusts the penalty weight on safety violations, converting a constrained problem into an unconstrained saddle-point optimization. If the agent violates a constraint, the multiplier increases, forcing the policy to prioritize safety. This adaptive balancing act is central to algorithms like PPO-Lagrangian and TRPO-Lagrangian benchmarked in Safety Gym.

The ability to identify inputs that differ significantly from the training distribution. In Safety Gym, an agent encounters novel hazard configurations not seen during training. A robust policy must recognize these unfamiliar states and default to a conservative, safety-first behavior rather than extrapolating recklessly. This links to epistemic uncertainty estimation and anomaly detection in neural networks.