Domain Randomization Benchmark

The core of any benchmark is a highly configurable simulator that serves as the source domain. This environment must support the programmatic variation of key parameters across visual, physical, and task-specific dimensions. Common examples include:

• MuJoCo or PyBullet for robotic manipulation tasks.
• Unity or Unreal Engine with the Robotics Operating System (ROS) bridge for high-fidelity vision-based tasks.
• Isaac Sim for GPU-accelerated, large-scale parallel simulation. The simulator's API must allow for on-the-fly parameter randomization during training episode rollouts.

A benchmark explicitly defines the set of parameters to be randomized and their statistical distributions. This creates the "reality gap" the model must overcome. Key categories include:

• Visual Parameters: Object textures, lighting (intensity, direction, color), camera properties (noise, distortion, position).
• Dynamics Parameters: Mass, friction, damping, actuator latency and gain, motor torque limits.
• Scene Configuration: Object spawn positions, number of distractors, background elements. The benchmark specifies whether randomization is uniform, follows a Gaussian distribution, or uses a more complex curriculum schedule.

To measure sim-to-real performance, the benchmark includes a corresponding set of real-world physical tasks or a held-out, high-fidelity validation domain. This is the ultimate test of generalization. Components include:

• Precise task definitions with success criteria (e.g., block insertion tolerance < 2mm).
• Standardized real-world setup (e.g., specific robot arm, camera, lighting rig) to ensure reproducibility across research labs.
• Protocols for zero-shot transfer, where the model is deployed on physical hardware without any fine-tuning on real data, providing the cleanest measure of DR efficacy.

Benchmarks provide a standardized scoring system to compare different DR methods. Metrics are task-specific but designed to be objective and repeatable. Common metrics include:

• Task Success Rate: Percentage of trials where the agent completes the objective.
• Average Reward / Return: For reinforcement learning agents, the cumulative reward achieved in the real world.
• Sim2Real Gap: The relative performance drop from simulation (where the model was trained) to reality.
• Robustness Score: Performance measured across a range of deliberately varied real-world conditions (e.g., different lighting, object wear) to test for over-randomization or brittleness.

To ground comparisons, a benchmark provides reference implementations of standard algorithms. This includes:

• Non-randomized simulation-trained baselines, which illustrate the reality gap.
• Manually-tuned Domain Randomization with a fixed parameter distribution.
• Advanced methods like Automatic Domain Randomization (ADR) or Randomized-to-Canonical networks. A public leaderboard tracks the performance of submitted methods, fostering competition and tracking progress in the field. Examples include benchmarks derived from the MetaWorld or RoboSuite frameworks.

Beyond a single score, effective benchmarks include tools for failure analysis and ablation studies. These help engineers understand why a method succeeds or fails:

• Parameter Sensitivity Analysis: Tools to test which randomized parameters (e.g., lighting vs. friction) most impact real-world performance.
• Visualization of Invariant Features: Methods to inspect what the model's perception pipeline focuses on, verifying invariant feature learning.
• Controlled Reality Gap Progression: Ability to gradually increase simulation fidelity to diagnose at what point the model fails, isolating the source of the domain gap.

Domain Randomization is the core technique of varying simulation parameters—like lighting, textures, and physics—across a wide range during training. This forces a model to learn invariant features and policies that generalize to the real world, compensating for the simulation-to-reality gap. It is the foundational method evaluated by Domain Randomization Benchmarks.

Sim-to-Real Transfer is the overarching objective: deploying a model trained in simulation to perform effectively on a physical robot or system. A Domain Randomization Benchmark measures the success of this transfer. The process involves:

• Training exclusively in a randomized simulator.
• Measuring the resulting real-world task performance (e.g., success rate, accuracy).
• The benchmark quantifies how well different DR methods achieve zero-shot sim-to-real transfer.

Automatic Domain Randomization is an advanced, algorithmic approach to DR. Instead of manually defining parameter ranges, ADR uses a search process (e.g., based on policy performance) to automatically discover and apply the most effective randomization settings. Benchmarks often compare traditional DR against ADR to evaluate which method yields better robust policy learning and generalization.

Visual Domain Randomization focuses specifically on randomizing perceptual inputs. This is a key component tested in vision-based benchmarks. Parameters include:

• Object textures and colors
• Lighting conditions (direction, intensity, color)
• Camera properties (noise, distortion, position)
• Background scenes The goal is to train vision models that are robust to the vast appearance variations encountered in reality.

Dynamics Randomization varies the physical parameters of a simulation to train policies robust to real-world physics uncertainty. This is critical for robotic control benchmarks. Randomized parameters include:

• Mass and inertia of objects and robot links
• Friction coefficients (sliding, torsional)
• Motor dynamics (latency, torque limits)
• Actuator strength and damping This teaches policies to adapt to different hardware wear, payloads, and surface conditions.

The Reality Gap (or Domain Gap) is the fundamental challenge: the discrepancy between the simulated training environment and the real world. It arises from:

• Unmodeled physics (e.g., air resistance, cable dynamics)
• Perceptual differences (sensor noise, render artifacts)
• Simplified actuation models A Domain Randomization Benchmark's primary purpose is to measure how effectively a given DR method bridges this gap, as quantified by the drop in performance from sim to real (Sim2Real Performance).