Curriculum Randomization

The core mechanism of curriculum randomization is a scheduler that controls the randomization distribution over time. Instead of sampling parameters from a fixed, wide range from the start, the scheduler begins with a narrow, easy distribution (e.g., low friction, simple textures). The range is gradually expanded according to a schedule, which can be:

• Linear Expansion: The bounds of uniform distributions increase linearly with training steps.
• Exponential Expansion: Parameters are sampled from distributions whose variance grows exponentially.
• Performance-Based: The curriculum advances only when the agent's performance (e.g., success rate, reward) in the current distribution exceeds a threshold, ensuring mastery before increasing difficulty.

Effective curriculum design requires structuring the parameter space into learnable segments. This involves:

• Independent vs. Coupled Parameters: Deciding whether to increase the range of parameters (like object mass and friction) independently or in a correlated manner that reflects physical plausibility.
• Difficulty Metrics: Defining a quantitative measure of "difficulty" for a given parameter set. For a grasping task, this could be the object's mass-to-friction ratio.
• Ordering Heuristics: Determining the sequence for introducing complexity. A common heuristic is to first randomize physics parameters (dynamics), then visual randomization (appearance), and finally sensor noise randomization, as physical interaction is often foundational.

Curriculum randomization is implemented within a reinforcement learning training loop. The policy (e.g., a neural network) is updated using experiences collected from a simulation environment whose parameters are controlled by the curriculum scheduler.

• On-Policy Algorithms: Like PPO, where the policy collects rollouts in the current randomized environment, then updates before the environment may change.
• Off-Policy Algorithms: Like SAC, which can learn from a replay buffer containing experiences from multiple difficulty levels, though this may require careful balancing.
• Reward Shaping: The reward function may need adjustment across the curriculum. Easier initial stages might use denser reward signals to guide learning, which are phased out as the task becomes harder.

A key challenge is ensuring the policy retains skills learned in easier domains as difficulty increases. Mitigation strategies include:

• Replay Buffers: Maintaining a buffer of experiences from all past difficulty levels and sampling from it during policy updates.
• Elastic Weight Consolidation: Applying regularization that penalizes changes to network weights important for previously mastered tasks.
• Gated Architectures: Using policy networks with conditioning inputs or modular components that can specialize for different parameter ranges without interfering.
• Cyclical Scheduling: Periodically returning to easier settings to reinforce foundational behaviors, preventing the policy from overfitting to the current hardest scenario.

Instead of a manually designed schedule, the curriculum can be learned automatically. This aligns with Automatic Domain Randomization (ADR) principles.

• Self-Paced Learning: The agent itself generates and selects training scenarios based on its current competence, often measured by learning progress.
• Goal Generation: In goal-conditioned tasks, an adversarial network or generator proposes increasingly challenging goals (parameter sets) for the policy to solve.
• Domain Complexity Estimation: Using a separate model to predict the failure probability of the current policy for a given parameter set, then prioritizing training on domains with moderate predicted difficulty (the "zone of proximal development").

Evaluating a curriculum randomization strategy requires specific protocols beyond final sim2real success rate.

• Interpolation and Extrapolation Tests: Test the policy on parameter values within the final trained range (interpolation) and beyond it (extrapolation) to assess generalization.
• Curriculum Ablation: Compare the final policy's robustness against one trained with static domain randomization from the start. The curriculum-trained policy should achieve comparable or better out-of-distribution (OOD) robustness with fewer total training samples.
• Sensitivity Analysis: Measure how sensitive the final performance is to the specific schedule (e.g., growth rate, performance thresholds) to determine the technique's stability.

A robotic arm learning a peg-in-hole task starts training in a simulation with narrow parameter bounds (e.g., object mass: 0.9–1.1 kg, friction coefficient: 0.7–0.9). The curriculum scheduler monitors success rate over a moving window. Once a threshold (e.g., 95% success) is consistently met, the randomization distribution is expanded (e.g., mass: 0.5–1.5 kg, friction: 0.3–1.2). This forces the policy to adapt to greater physical uncertainty incrementally, preventing early training collapse from excessive difficulty.

For a vision-based mobile robot, curriculum randomization manages visual domain shift. Initial training uses a simple, uniform visual style: basic textures, consistent studio lighting. As the policy masters navigation, the randomization curriculum introduces:

• Lighting variations (direction, color temperature, shadows)
• Texture randomization on floors and walls
• Distractor objects with randomized shapes and colors
• Camera artifact simulation (blur, noise, exposure variance) This staged approach builds a perception system robust to the visual noise of real warehouses or homes.

A legged robot learning to walk employs a terrain difficulty curriculum. Training begins on flat planes with minor roughness. The curriculum algorithm, often driven by a learning progress signal, gradually introduces more challenging terrains:

• Sloped surfaces with increasing incline
• Randomly placed obstacles (pebbles, steps)
• Deformable materials like grass or mud simulants
• Unstable platforms with moving sections By controlling the probability distribution over terrain types, the policy learns basic balance before mastering complex locomotion, improving sample efficiency and stability.

To bridge the reality gap in sensor readings, curriculum randomization manages synthetic noise injection. A policy relying on LiDAR or joint encoders starts training with minimal or no synthetic noise. The curriculum progressively adds bounded noise models:

• Gaussian noise with increasing standard deviation
• Dropout noise simulating sporadic sensor failures
• Quantization error mimicking lower-resolution sensors
• Temporal latency in state updates This method ensures the policy does not become reliant on perfect simulation data and can handle the imperfect sensing of real hardware.

Simulated actuators are often idealized. Curriculum randomization hardens policies against real-world actuator dynamics by slowly varying:

• Motor torque limits and saturation models
• Gear backlash and compliance
• Control delay in the command pipeline
• Battery voltage drop-off effects on power Training might start with high-torque, instantaneous actuators. The curriculum then reduces torque limits and introduces latency and non-linearities, forcing the control policy to learn robust, energy-efficient motions that transfer to physical motors.

Automatic Domain Randomization (ADR) is an algorithmic implementation of curriculum randomization. It starts with a single, easy simulation domain. A distinguisher network tries to detect which domain generated a rollout. If the distinguisher succeeds, the parameter space is automatically expanded in that direction, creating a new, harder domain. This creates a self-play curriculum where the policy is constantly challenged at the frontier of its capabilities, efficiently exploring the randomization space without manual schedule tuning.

Domain Randomization (DR) is the foundational technique upon which curriculum randomization builds. It involves training a model in a simulation where environmental parameters—such as object textures, lighting, physics properties (mass, friction), and sensor noise—are randomly varied across episodes. The core hypothesis is that by exposing the policy to a vast, randomized distribution of simulated worlds, it will learn features invariant to these variations and generalize zero-shot to the real world. This is a primary method for bridging the reality gap.

Automatic Domain Randomization (ADR) is an algorithmic extension that automates the expansion of the randomization range. Instead of a human-defined, static distribution, ADR starts with an easy setting and progressively generates more challenging simulation environments as the policy learns. It automatically discovers the boundaries of the parameter space where the policy fails and pushes randomization toward those worst-case domains. This creates a curriculum driven by the policy's own performance, making it a close sibling to curriculum randomization.

Bounded Randomization defines the constraints within which parameters are varied during training. It is a critical design choice for both standard and curriculum-based DR. The bounds ensure randomized parameters remain within physically plausible or safe limits, preventing the policy from learning in unrealistic or detrimental scenarios. In curriculum randomization, these bounds are typically expanded over time, starting with tight, easy bounds (e.g., low friction, simple lighting) and progressing to wider, more challenging ones that better approximate real-world variability.

The Randomization Distribution is the probability distribution from which simulation parameters are sampled during each training episode. Common choices include uniform, Gaussian (normal), or log-normal distributions. In curriculum randomization, the parameters of this distribution itself are often what is progressively altered. For example, training might start by sampling friction from a narrow uniform distribution U(0.2, 0.4) and gradually expand to U(0.1, 1.0). The choice of distribution and its evolving parameters directly controls the difficulty curriculum.

Policy Conditioning is a technique where the current domain's randomized parameters (the domain parameters) are provided as an additional input to the policy or value network. This allows the network to explicitly adapt its behavior based on the perceived characteristics of the environment. In curriculum randomization, policy conditioning can be crucial. As the difficulty increases, the policy can use these conditioning signals to modulate its actions, effectively learning a family of behaviors for different domains rather than a single, overly averaged policy.

The Simulation Fidelity Trade-off is a key engineering consideration that interacts with curriculum design. High-fidelity simulations are computationally expensive but more accurately model reality. Low-fidelity simulations are fast but have a larger reality gap. Curriculum randomization often operates effectively in lower-fidelity simulators because the extensive randomization compensates for inaccuracies. The curriculum can be designed to start in a simpler, faster simulation mode and gradually introduce more computationally intensive, higher-fidelity elements (like precise contact dynamics) as training progresses.