Randomization Distribution

The choice of probability distribution determines how parameters are varied. Common types include:

• Uniform Distribution: Parameters are sampled with equal probability within a defined minimum and maximum range. Simple and effective for bounded parameters like object color or lighting intensity.
• Gaussian (Normal) Distribution: Parameters are sampled from a bell curve defined by a mean (μ) and standard deviation (σ). Useful for modeling parameters with natural variation around a nominal value, like sensor bias.
• Log-Normal Distribution: Used for parameters that must remain strictly positive (e.g., mass, friction coefficients), ensuring samples are never zero or negative.
• Categorical Distribution: Used for discrete, non-numeric parameters, such as selecting between different object textures or background scenes.

The parameter space is the complete set of all simulation variables subject to randomization. It is the Cartesian product of the ranges for each individual parameter. Key aspects include:

• Dimensionality: The number of randomized parameters. High dimensionality increases the coverage of possible environments but can make training more challenging.
• Interdependence: Some parameters may be correlated (e.g., daytime lighting intensity and sun position). Distributions can be defined jointly to preserve these physical relationships.
• Bounded vs. Unbounded: Bounded randomization keeps parameters within physically plausible limits to prevent training on nonsensical scenarios that could harm policy learning.

This governs when and how often new parameters are drawn from the distribution during training.

• Per-Episode Sampling: New parameters are sampled at the start of each training episode and held constant throughout. This is the most common strategy, teaching the policy to complete a task under a fixed, randomized condition.
• Per-Step Sampling: Parameters are re-sampled at every simulation timestep. This creates an extremely dynamic and challenging environment, often used to enforce strict invariance to certain conditions.
• Curriculum Sampling: The distribution's properties (e.g., the range of a uniform distribution) are gradually expanded over time, starting with easier, narrower settings and progressing to harder, broader ones.

A core design tension exists between maximizing the distribution's coverage and ensuring it remains anchored to reality.

• Wide Coverage: Broad distributions increase the diversity of training experiences, promoting out-of-distribution (OOD) robustness and helping the policy generalize to unseen real-world conditions.
• Reality Anchor: If distributions are too broad or unrealistic, the policy may learn to exploit simulation artifacts or develop strategies that fail on physical hardware. The distribution should be centered around a nominal domain estimated via system identification.
• Automatic Domain Randomization (ADR) algorithms dynamically adjust distribution bounds during training to find an optimal balance, expanding the range until the policy's performance begins to degrade.

Advanced randomization schemes move beyond simple independent sampling.

• Policy Conditioning: The sampled parameters are fed as an additional input vector to the policy network. This allows the policy to explicitly adapt its strategy to the current domain parameters, rather than learning a single invariant policy.
• Contextual Distributions: The distribution for one parameter can be conditioned on the value of another. For example, the distribution for 'ground friction' could have a higher mean if the 'surface type' parameter is sampled as 'ice'.
• Worst-Case Domain Sampling: Instead of random sampling, training can focus on parameters from the distribution that constitute a worst-case domain for the current policy, actively seeking out and improving upon failure modes.

The choice of randomization distribution is intrinsically linked to the underlying simulation fidelity.

• Low-Fidelity Simulators: Often rely heavily on aggressive randomization (e.g., large visual variations, broad physics ranges) to compensate for their inherent inaccuracies.
• High-Fidelity Simulators: Can use narrower, more targeted randomization distributions because the base simulation is more accurate. Randomization here often focuses on real-world stochasticity (e.g., sensor noise, slight material property variations) rather than compensating for large systematic errors.
• The Fidelity-Randomization Trade-off: Engineers balance computational cost. A cheaper, lower-fidelity simulator with broader randomization can sometimes achieve better sim2real success rates than an expensive, high-fidelity simulator with poor or no randomization.

The Uniform Distribution is the most common choice for domain randomization, where each parameter value within a defined interval [a, b] is equally likely to be sampled. It is favored for its simplicity and for imposing a hard prior that the policy must perform well across the entire range.

• Use Case: Randomizing object masses, friction coefficients, or lighting intensities where no single value is more probable than another.
• Example: Sampling a robot's joint damping coefficient uniformly between 0.5 and 5.0 N·m·s/rad.
• Consideration: Can be inefficient if the real-world parameter is known to follow a different distribution, as it wastes capacity on unrealistic extremes.

The Gaussian Distribution samples parameters according to a bell curve defined by a mean (μ) and standard deviation (σ). It is used when a parameter's real-world value is expected to cluster around a nominal point with some natural variation.

• Use Case: Modeling sensor noise (e.g., Gaussian noise added to camera images or joint position readings) or variations in material properties around a known average.
• Example: Sampling the coefficient of restitution for a rubber ball with μ = 0.85 and σ = 0.05.
• Consideration: Requires careful selection of σ; too small fails to provide sufficient diversity, while too large can generate physically implausible outliers.

The Log-Normal Distribution is used for parameters that are inherently positive and have a right-skewed distribution, where values can vary by orders of magnitude. If a parameter's logarithm is normally distributed, the parameter itself follows a log-normal distribution.

• Use Case: Randomizing time delays, actuator response times, or lighting intensities in real-world scenes, which often follow multiplicative, non-negative processes.
• Example: Sampling the latency of a simulated network controller, where most values are low but occasional significant delays occur.
• Benefit: Naturally prevents sampling non-physical negative values and better models real-world heavy-tailed phenomena.

The Beta Distribution is a flexible family of distributions defined on the interval [0, 1], parameterized by two shape parameters (α, β). It is ideal for randomizing probabilities, ratios, or normalized parameters.

• Use Case: Varying the probability of an event occurring in simulation (e.g., sensor dropout rate) or the mix of materials in a composite object.
• Example: Sampling the opacity of a semi-transparent object, where α=2, β=5 creates a distribution skewed towards lower opacity.
• Advantage: Its bounded support and shape flexibility allow precise control over the likelihood of extreme values (near 0 or 1).

The Categorical Distribution is used for discrete, non-ordinal parameters where the simulation must switch between distinct states or categories. It assigns a probability to each possible category.

• Use Case: Randomizing object textures, robot gripper types, background scenes, or the presence/absence of environmental features.
• Example: A simulation for a warehouse robot randomly selects a box texture from a set of 10 options, each with equal probability.
• Implementation: Often combined with continuous distributions; e.g., a categorical choice of object shape paired with a uniform distribution for its size.

A Mixture Distribution combines multiple simpler distributions (components) to model complex, multi-modal parameter spaces. The overall distribution is a weighted sum of its components.

• Use Case: Modeling environments where a parameter can come from distinct regimes. For example, floor friction could be low (ice), medium (wood), or high (rubber), each modeled by its own distribution.
• Example: Sampling lighting conditions from a mixture of a distribution for 'daytime' (bright, uniform) and 'night' (dark, spotty).
• Power: Enables the creation of rich, structured randomization spaces that better approximate the disjoint modes of real-world variation, leading to more comprehensive robustness.

In domain randomization, the parameter space defines the complete set of all simulation variables that can be varied during training. This includes:

• Physical dynamics (mass, friction, motor torque)
• Visual attributes (textures, lighting, camera pose)
• Sensor models (noise type, latency, dropout)

The randomization distribution is applied over this space. A well-designed parameter space is crucial; it must be broad enough to cover real-world variations but constrained enough to allow for efficient learning. The choice of which parameters to randomize is a key engineering decision.

Bounded randomization is the practice of constraining the variation of simulation parameters within specific, physically plausible limits. Instead of sampling from an unbounded distribution (e.g., a Gaussian with infinite support), parameters are sampled from a bounded distribution like a uniform or truncated normal distribution.

This ensures the training environment remains within a realistic domain, preventing the policy from learning from nonsensical physics (e.g., negative mass) or visual conditions that would never occur in reality. It directly shapes the randomization distribution to improve sample efficiency and final robustness.

Curriculum randomization is a training strategy that progressively increases the difficulty or range of randomized parameters. It starts with a narrow randomization distribution (e.g., low variance) or easier task settings, and gradually expands it as the policy learns.

This technique helps overcome the exploration challenge in highly randomized environments. By providing a learning progression, it can lead to more stable training and ultimately a policy that is robust to a wider range of conditions than if trained on the full distribution from the start. It's an adaptive approach to managing the parameter space.

A randomized simulation ensemble involves training a single policy by exposing it to multiple parallel simulation instances, each with different randomized parameters sampled from the randomization distribution. This is a common implementation pattern in modern RL frameworks.

Instead of sequentially experiencing varied conditions, the policy learns from a diverse batch of environments simultaneously. This massively parallel approach is key to making domain randomization computationally feasible, as it amortizes the cost of environment resets and provides a rich, mixed gradient signal that encourages robust, generalized behavior.

In robust optimization for sim-to-real transfer, the worst-case domain refers to the specific set of simulation parameters (within the defined randomization range) that most challenges the policy's performance. It represents the hardest conditions the policy might face.

Training explicitly to perform well under this adversarial configuration is a more rigorous goal than average performance. Techniques like adversarial domain randomization actively search for these hard parameters during training to stress-test and strengthen the policy. Identifying the worst-case domain is also critical for safety validation before real-world deployment.

Viewing domain randomization through the lens of data augmentation reframes it as a technique to synthetically expand the training dataset. Each set of randomized parameters generates a novel, but semantically consistent, training example.

For perception tasks, varying textures and lighting is directly analogous to image augmentation. For control, varying physics parameters augments the dynamics experience of the agent. This perspective connects domain randomization to broader domain generalization literature in machine learning, where the goal is to learn invariances to nuisance parameters. The randomization distribution effectively defines the augmentation policy.