Automatic Domain Randomization (ADR)

ADR's core mechanism is a self-adjusting curriculum that eliminates the need for manual tuning of randomization ranges. The system starts with a narrow, easy distribution of simulation parameters (e.g., low friction, simple lighting). As the policy masters these conditions, the algorithm automatically expands the parameter distributions, creating progressively harder environments. This ensures the policy is continuously challenged, driving it toward robust, domain-invariant solutions.

The expansion of the randomization domain is directly tied to the policy's success rate. A common implementation uses a threshold-based rule: if the policy's performance exceeds a target success rate over recent episodes, the system widens the bounds of one or more randomized parameters (e.g., increasing the maximum object mass or the range of camera angles). This creates a direct feedback loop where policy improvement triggers environmental complexity, preventing the agent from overfitting to a static set of randomized conditions.

While standard domain randomization often focuses on visual properties, ADR is particularly effective for dynamics randomization. It can automatically expand the ranges of physical parameters that are critical for real-world transfer but difficult to model accurately, such as:

• Actuator dynamics (motor gain, latency, backlash)
• Contact physics (friction coefficients, restitution)
• Mass and inertia of robotic links and manipulated objects
• Sensor noise models for proprioception and vision By systematically making the physics more diverse and challenging, ADR forces the policy to learn fundamental physical principles rather than exploiting simulation artifacts.

ADR does not simply make parameters more extreme; it constructs a path of increasing complexity through the domain space. This progression can be multi-dimensional, affecting different parameter groups at different times. For example, the system might first expand lighting and texture ranges, then once mastered, begin randomizing object sizes and shapes, and finally tackle the most difficult physical dynamics. This structured approach is more sample-efficient than uniform random sampling across all parameters from the start.

A key failure mode in sim-to-real transfer is overfitting to the simulator's specific physics engine or rendering pipeline. ADR directly combats this by ensuring the policy never settles into a local optimum for a fixed simulation configuration. By constantly shifting the ground truth of the environment, the policy is incentivized to find strategies that work across a broad, continuous spectrum of possible worlds, making it more likely to generalize to the singular, unseen world of reality.

ADR can be viewed as an unsupervised domain adaptation technique performed during training. Instead of aligning feature distributions between a fixed source and target, it continuously expands the source domain to envelop the target. The theoretical goal is to create a source distribution so broad that the real-world target distribution becomes a probable sample from it. This aligns with the concept of covering the reality gap with a sufficiently diverse set of simulated experiences, making zero-shot transfer more feasible.

Unlike standard Domain Randomization, which uses a fixed, pre-defined range of parameters, ADR dynamically expands the randomization range. This creates an automatic curriculum.

• Standard DR: Manually set bounds (e.g., friction between 0.2 and 0.8). The policy may overfit to this range.
• ADR: Starts with narrow bounds. When the policy succeeds, a domain critic expands the bounds (e.g., friction now 0.1 to 0.9), generating new, harder environments.
• Result: ADR systematically explores the parameter space, often leading to more robust policies and reducing the need for manual tuning of randomization ranges.

The ADR process is driven by specific, interconnected components that automate the curriculum creation.

• Domain Parameter Space: The set of simulation parameters to be randomized (e.g., physics properties, visuals).
• Policy (Actor): The neural network being trained via reinforcement learning.
• Domain Critic: A learned model or heuristic that evaluates policy performance and decides when and how to expand randomization ranges.
• Automatic Boundary Expansion: The mechanism that increases the range or changes the distribution of a randomized parameter (e.g., uniformly expanding min/max values).
• Performance Threshold: A success metric (e.g., average reward) that triggers the domain critic to make the environment more challenging.

The foundational technique upon which ADR builds. Domain Randomization involves training a policy in a simulation where a fixed, pre-defined set of parameters—such as object textures, lighting conditions, camera angles, and physics properties like mass and friction—are randomly sampled from a static distribution. The goal is to force the policy to learn domain-invariant features that generalize to unseen real-world conditions, rather than overfitting to the specifics of a single simulation.

A critical subset of domain randomization focused specifically on physical parameters. Dynamics Randomization involves varying the fundamental physics engine constants of a simulation, including:

• Mass and inertia of bodies
• Joint damping and friction coefficients
• Motor torque limits and latency
• Gravity and air resistance By randomizing these core dynamics, policies become robust to the inevitable inaccuracies in simulated physics and the variations found in real-world hardware, such as wear on motors or uneven floor surfaces.

The fundamental problem that ADR and related methods aim to solve. The Reality Gap is the performance degradation observed when a policy trained in simulation fails upon deployment in the physical world. This gap is caused by simulation-to-reality discrepancies in:

• Visual rendering (textures, lighting, sensor noise)
• Physical dynamics (contact modeling, actuator response)
• Unmodeled effects (vibration, air currents, cable dynamics) ADR directly attacks this gap by automatically expanding the simulation's parameter space to envelop the real-world conditions.

A complementary approach to randomization for closing the reality gap. System Identification is the process of building or refining a mathematical model of a physical system (e.g., a robot's arm dynamics) using data collected from the real hardware. Instead of randomizing parameters broadly, SysID seeks to precisely calibrate the simulation to match real-world measurements. This high-fidelity model can then be used for more sample-efficient training or combined with ADR, where the identified parameters form the center of the randomization distribution.

A deployment-phase strategy often used alongside simulation-trained policies. Online Adaptation refers to a policy's ability to continuously adjust its parameters or internal state in real-time during operation on the physical system. While ADR creates robustness before deployment, online adaptation handles residual uncertainties and unforeseen changes in the environment. Techniques range from fine-tuning neural network weights to updating the parameters of a probabilistic dynamics model, allowing the system to compensate for drift, payload changes, or damage.

The ideal outcome of robust sim-to-real training methods like ADR. Zero-Shot Transfer is the successful deployment of a simulation-trained policy onto a physical robot without any additional training or fine-tuning on real-world data. The policy must generalize perfectly from its varied simulated experiences to the novel real domain. Achieving reliable zero-shot transfer is the primary benchmark for evaluating the effectiveness of domain randomization and ADR, as it eliminates the need for costly and potentially dangerous real-world data collection.