Domain Randomization

The systematic variation of simulation parameters to create a diverse training distribution. This is the foundational mechanism of Domain Randomization.

Key parameters include:

• Visual Properties: Object textures, colors, lighting conditions, camera noise, and background scenes.
• Physical Properties: Object mass, friction coefficients, damping, and motor strength.
• Environmental Dynamics: Gravity, air resistance, and sensor latency.

By training across this broad, randomized parameter space, the learned policy becomes invariant to specific simulation details and generalizes to unseen real-world conditions.

Prioritizes learning a policy that works under many inaccurate simulations over one that is perfect for a single, high-fidelity simulation. This principle acknowledges that creating a photorealistic, physically exact simulator is often impossible or prohibitively expensive.

The strategy is to intentionally introduce inaccuracies (e.g., unrealistic colors, simplified physics) in a controlled, randomized way. The policy must then rely on invariant features of the task (e.g., geometric relationships, functional object properties) rather than brittle, simulation-specific cues. This builds a form of intrinsic robustness that bridges the sim-to-real gap.

The training goal is to find a single policy that maximizes expected return across the entire distribution of randomized simulation instances, not just a single environment.

Mathematically, this is framed as: θ* = argmax_θ E_{τ ~ p(τ), s ~ p(s|τ)} [ Σ R(s, a) ] Where τ represents a sampled set of randomized parameters (a "domain") and s are states within that domain.

This expectation over randomized domains forces the policy's parameters θ to perform well everywhere within the support of the randomization, making it robust to the parameter variations it will encounter in reality.

Advanced Domain Randomization often employs a curriculum or automatic tuning of the randomization ranges to improve learning efficiency and final performance.

Common approaches:

• Adaptive Domain Randomization (ADR): Automatically expands the randomization range for parameters where the policy is already successful, creating a progressively harder curriculum.
• Bayesian Optimization: Searches for a distribution of parameters that is "just hard enough" to facilitate learning without being impossible.
• Task-Relevant Randomization: Focuses variation on parameters that most affect task performance (e.g., friction for pushing, object dimensions for grasping), rather than randomizing everything uniformly.

This moves beyond naive uniform randomization towards intelligent, data-driven sampling.

A specialized variant where parameters related to the robot's own dynamics are randomized, not just the environment. This is critical for dexterous manipulation where precise force control is needed.

Randomized system parameters include:

• Actuator gains, latency, and backlash.
• Link masses and inertias.
• Joint friction and damping.
• Tactile sensor response models.
• Payload mass (for unknown objects in-hand).

By training with a wide range of possible self-models, the policy learns to be adaptive to its own perceived dynamics, compensating for the inevitable inaccuracies in the real robot's calibrated model. This is key for contact-rich tasks like in-hand manipulation and tactile servoing.

Domain Randomization is powerful but has known limitations, leading to its use alongside other sim-to-real methods.

Key Limitations:

• Can fail if the real world contains phenomena completely absent from the randomization distribution (e.g., a new object type).
• May learn overly conservative policies if randomization is too extreme.
• Does not actively learn a model of the real world.

Complementary Techniques:

• Domain Adaptation: Actively aligns simulation and real-world data distributions using learned transformations.
• System Identification: Uses real-world data to calibrate simulation parameters before or during deployment.
• Meta-Learning: Trains a policy that can quickly adapt to a new, identified domain with few real-world trials.

In practice, a hybrid approach is often most effective.

Randomizing visual properties forces the policy to rely on geometric and structural features rather than superficial textures or colors.

• Textures & Materials: Object and background surface properties (e.g., wood, metal, plastic, random colors, patterns).
• Lighting Conditions: Number, type, color, intensity, and position of light sources; ambient light levels.
• Camera Properties: Field of view, focal length, sensor noise, resolution, and lens distortion.
• Object Colors & Patterns: Applying random RGB values or image textures to object meshes.

Varying physical parameters encourages the policy to adapt to different object behaviors and interaction forces.

• Mass & Inertia: Randomizing the mass and inertia tensors of objects and robot links.
• Friction Coefficients: Varying static and dynamic friction between contacting surfaces.
• Damping & Restitution: Adjusting joint damping (resistance) and object restitution (bounciness).
• Actuator Dynamics: Modifying motor strength, control latency, and torque limits.

Altering the layout and composition of the environment prevents the policy from memorizing a single setup.

• Object Count & Placement: Randomizing the number, type, and initial positions of objects in the scene.
• Background & Distractors: Adding random 3D models or 2D image planes as visual clutter.
• Workspace Bounds: Changing the size and shape of the allowable robot operating area.
• Initial State Randomization: Varying the starting pose of the robot and target objects for each training episode.

Injecting noise and distortion mimics imperfections in real-world sensors, making the policy robust to measurement errors.

• Additive Noise: Adding Gaussian or uniform noise to proprioceptive (joint) and exteroceptive (camera, depth) readings.
• Quantization & Dropout: Simulating sensor resolution limits or temporary signal loss.
• Delay & Latency: Introducing stochastic delays between state observation and action execution.
• Calibration Errors: Applying small, random offsets to sensor readings to mimic miscalibration.

Tailoring randomization to the specific task and environment is critical for effective transfer.

• For Dexterous Manipulation: Randomize object dimensions, grasp stiffness, and finger pad friction.
• For Autonomous Driving: Vary time of day, weather effects (rain, fog), and traffic density.
• For Drone Navigation: Randomize wind gusts, atmospheric density, and GPS signal noise.
• For Industrial Robotics: Vary conveyor belt speed, part orientation jitter, and ambient vibrations.

The strategy for varying parameters over time can significantly impact training efficiency and final robustness.

• Uniform Randomization: Parameters are sampled from a fixed, wide distribution every episode.
• Curriculum Randomization: The range of parameters is gradually increased from a narrow, easy setting to the full target distribution.
• Adversarial Randomization: A second network actively searches for parameter settings that challenge the policy most.
• Systematic Sweeping: Exhaustively testing across a grid of discrete parameter values rather than random sampling.

The sim-to-real gap is the performance discrepancy between a robot policy trained in simulation and its performance when deployed on physical hardware. This gap arises from model inaccuracies in the simulator's physics engine, rendering, and sensor models. Domain randomization directly targets this gap by exposing the policy to a vast distribution of simulated conditions, making it robust to the unmodeled dynamics and visual discrepancies of reality.

System identification is the alternative, model-based approach to sim-to-real transfer. Instead of randomizing simulation parameters, this method involves:

• Precisely measuring real-world physical properties (e.g., friction coefficients, motor dynamics, object masses).
• Calibrating the simulation to match these measured parameters as closely as possible.
• Training the policy in this calibrated, high-fidelity simulation. It is often more sample-efficient for narrow domains but requires extensive physical data collection and can be brittle if the real world drifts from the identified model.

Reinforcement Learning is the primary training paradigm used with domain randomization. An RL agent learns a policy by interacting with the randomized simulation, receiving rewards for successful task completion. The randomization creates a diverse set of environmental dynamics, forcing the RL algorithm to discover policies that are invariant to these perturbations. This makes the policy generalizable rather than overfit to a single, deterministic simulation instance.

Domain adaptation is a broader machine learning technique for transferring models from a source domain (simulation) to a target domain (reality). Unlike domain randomization, which acts during training, many domain adaptation methods work during inference or via fine-tuning. Key approaches include:

• Adversarial domain adaptation: Using a discriminator network to make features from simulation and reality indistinguishable.
• Fine-tuning on real data: Using a small amount of real-world experience to adapt the sim-trained policy. Domain randomization is considered a proactive, training-time form of domain adaptation.

Often used synonymously with the sim-to-real gap, the reality gap specifically emphasizes the causal disconnect between simulated and physical causality. A policy may exploit simulator artifacts (e.g., perfect collision detection, simplified friction models) to achieve high reward through actions that are ineffective or dangerous in reality. Domain randomization mitigates this by preventing the policy from relying on any consistent artifact, as the underlying physics and visuals are constantly changing.

Curriculum learning is a training strategy often combined with domain randomization. Instead of applying full randomization from the start, the training begins in a simple, deterministic simulation. The complexity is gradually increased by:

• Widening the randomization ranges (e.g., starting with small lighting variations, progressing to extreme ones).
• Adding new randomized parameters over time. This curriculum helps the learning algorithm stabilize and converge more effectively than facing the full complexity of randomization immediately.