Curriculum Learning

The core of curriculum learning is defining a difficulty metric and a scheduler. The metric quantifies task complexity (e.g., object density, target distance, force required). The scheduler determines when to advance to the next task based on agent performance.

• Common Metrics: Success rate, reward magnitude, or variance in agent actions.
• Scheduler Types: Linear (pre-defined steps), adaptive (advances upon reaching a performance threshold), or reverse (starts hard and adds simplifications).
• Example: Training a robotic arm to grasp might start with large, fixed objects in an empty scene (easy) and progress to small, cluttered objects on a moving conveyor (hard).

Instead of a hand-designed curriculum, the task sequence can be generated automatically by the learning process itself. This is crucial for complex sim-to-real domains where the optimal path is unknown.

• Goal-Based: The agent or a teacher algorithm proposes increasingly challenging goals (GoalGAN).
• Adversarial: A second network generates tasks that are at the current limit of the agent's capability.
• Self-Paced Learning: The agent samples from a distribution of tasks, adjusting the distribution towards harder tasks as its competence improves. This creates a smooth learning trajectory that maximizes sample efficiency.

Curriculum learning is often combined with Domain Randomization. Instead of randomizing all parameters from the start, the curriculum gradually increases the randomization range.

• Initial Phase: Train in a narrow, deterministic simulation to learn basic skills.
• Progressive Phase: Systematically increase variance in physics parameters (mass, friction), visuals (textures, lighting), and sensor noise.
• Outcome: The policy develops robustness in a controlled manner, reducing the risk of catastrophic forgetting that can occur with full randomization from the outset. This hybrid approach is a best practice for sim-to-real transfer.

A key challenge is catastrophic forgetting, where learning new, harder tasks degrades performance on earlier, mastered ones. Curriculum design must manage this plasticity.

• Techniques: Use experience replay buffers that store data from all difficulty levels, or employ elastic weight consolidation to penalize changes to weights important for previous tasks.
• The Stability-Plasticity Dilemma: The curriculum must balance retaining old skills (stability) with acquiring new ones (plasticity).
• Monitoring: Track performance on a validation set of tasks from all difficulty levels to detect forgetting.

In robotics, curriculum learning directly addresses the reality gap by decomposing the transfer problem.

1. Skill Acquisition in Sim: Learn fundamental motor skills (e.g., stable walking) in a high-fidelity but noiseless simulation.
2. Robustness Training: Introduce simulated disturbances (pushes, uneven terrain) as the 'next lesson'.
3. Sensorization: Progress from perfect state information to noisy, pixel-based observations.
4. Deployment: The final policy, hardened by this graduated exposure to complexity, exhibits higher zero-shot transfer success to physical hardware.

Measuring curriculum learning efficacy requires specific benchmarks beyond final task performance.

• Sample Efficiency: The total number of training steps or episodes required to reach a performance threshold.
• Asymptotic Performance: The final level of skill achieved compared to non-curriculum (flat) training.
• Transfer Gap: The difference in performance between the simulation validation environment and the final real-world test.
• Standardized Testbeds: Research often uses environments like Meta-World (robotic manipulation) or Procgen (procedural game environments) to compare curriculum strategies.

This is the core application in robotics. A curriculum is designed to incrementally close the reality gap.

• Stage 1: Train in a high-fidelity, deterministic simulation with perfect state information.
• Stage 2: Introduce domain randomization (varying textures, lighting, masses) to the simulation.
• Stage 3: Add sensor noise and latency to the simulated observations.
• Stage 4: Transfer to Hardware-in-the-Loop (HIL) testing with real actuators/sensors.
• Stage 5: Full deployment on the physical robot. This staged approach prevents the policy from overfitting to simplistic simulation dynamics and builds robustness step-by-step.

Used to teach robots complex manipulation tasks by breaking them into simpler sub-tasks.

• Example: Door Opening
  1. Learn to reach and touch the door handle in free space.
  2. Learn to grasp the handle firmly with randomized handle shapes.
  3. Learn to turn the handle against randomized friction.
  4. Learn to push the door open against randomized door weight and hinge stiffness.
• Example: Precision Insertion (Peg-in-Hole) The curriculum starts with a large hole and a tapered peg, progressively reducing the clearance and moving to a cylindrical peg, while also introducing positional uncertainty.

Training stable walking and running gaits for bipedal and quadrupedal robots is notoriously difficult. Curriculum learning mitigates this by controlling task difficulty.

• Initial State: The robot starts standing upright, close to a stable configuration.
• Gradual Progression: The curriculum slowly increases the commanded walking speed, introduces slope variations, and adds external push disturbances.
• Environment Complexity: Training progresses from flat planes to randomized terrain with small obstacles, then to stairs or rubble.
• Real-World Impact: Companies like Boston Dynamics utilize such curricula in simulation to bootstrap policies for robots like Atlas and Spot before real-world refinement.

Curriculum learning structures the challenge of navigating cluttered, dynamic environments.

• Static Environments First: Learn to map and navigate empty spaces, then static obstacle courses.
• Introduce Dynamics: Add slow-moving, predictable obstacles. Gradually increase obstacle speed and randomness.
• Multi-Agent Scenarios: For warehouse AMRs, start with a single robot, then introduce other agents with simple policies, finally scaling to complex multi-robot coordination.
• Perception Difficulty: Begin with perfect pose information, transition to raw egocentric perception (e.g., camera/LiDAR data) with increasing levels of sensor noise and occlusion.

The curriculum itself can be designed manually or learned automatically.

• Manual Sequencing: Expert defines the sequence of tasks/environments. Simple but requires domain knowledge.
• Self-Paced Learning: The agent itself gauges its competence (e.g., based on success rate) and decides when to advance to a harder task.
• Teacher-Student Frameworks: A separate "teacher" network learns to propose tasks that maximize the learning progress of the "student" policy. The teacher is often trained using Reinforcement Learning or Bayesian Optimization.
• Goal-Based Curricula: In goal-conditioned RL, the curriculum starts with easy-to-reach goals and progressively samples goals farther away or requiring more complex sequences of actions.

Curriculum learning is rarely used in isolation; it combines synergistically with other methods.

• With Domain Randomization: The curriculum can control the range of randomization. Start with narrow, realistic parameters and expand the randomization distribution as the policy becomes more robust.
• With Residual Policy Learning: Train a base policy in simulation using a curriculum. Deploy it on the real robot, then learn a small residual policy that corrects for the remaining reality gap. The residual policy's training can also follow a curriculum of correction magnitudes.
• With Imitation Learning: Use a curriculum to blend behavioral cloning from expert demonstrations (for initial safe exploration) with subsequent reinforcement learning fine-tuning on harder objectives.

A core sim-to-real technique where a policy is trained across a wide distribution of randomized simulation parameters (e.g., textures, lighting, friction coefficients, object masses). This forces the policy to learn robust, invariant features rather than overfitting to a single, imperfect simulation, thereby improving its chances of generalizing to the unseen real world. It is often used as a parallel or complementary strategy to Curriculum Learning.

The fundamental discrepancy between a simulation and the real world that Curriculum Learning aims to bridge. This gap manifests in:

• Dynamics: Inaccurate physics modeling (friction, contact, actuator lag).
• Perception: Differences in lighting, textures, and sensor noise.
• Actuation: Imperfect motor control and calibration errors. Curriculum Learning mitigates this by starting training in a simplified or more forgiving version of the simulation before gradually introducing complexity that better approximates reality.

The process of building or refining a mathematical model of a physical system's dynamics by observing its input-output behavior. In sim-to-real workflows, system identification is used to calibrate the simulation's physics parameters (like inertia or motor constants) to more closely match the real robot. A more accurate simulation, informed by system ID, provides a better foundation for Curriculum Learning, allowing the difficulty progression to be more meaningful and effective.

A technique where a learned neural network policy corrects the outputs of a traditional, analytically derived controller. The base controller (e.g., a PID or MPC) provides stable, safe, but potentially suboptimal behavior. The residual network learns to output adjustments to these commands. Curriculum Learning can be applied by first training the residual on easy tasks where the base controller is mostly correct, then progressively on harder scenarios requiring larger corrections.

A two-stage sim-to-real approach where a policy is pre-trained in simulation and then adapted using a limited amount of real-world data. Curriculum Learning is typically employed in the first, simulation-based stage to achieve high performance efficiently. The resulting policy serves as a strong initialization for the second-stage fine-tuning, significantly reducing the amount of costly and potentially risky real-world interaction needed.

The ability of a learned policy to maintain high performance despite variations in environmental conditions, sensor noise, or actuator dynamics. This is the primary objective of techniques like Curriculum Learning and Domain Randomization. A robust policy is essential for successful sim-to-real transfer. Curriculum Learning builds robustness progressively by exposing the agent to a controlled, expanding set of disturbances and task variations during training.