Curriculum Learning

The core of curriculum learning is defining a difficulty metric and a scheduler. The metric quantifies task hardness (e.g., success rate, reward sparsity, environmental complexity). The scheduler uses this to order tasks, typically starting with easy instances where reward signals are dense and the policy can learn basic skills before progressing to harder scenarios.

• Examples: Training a robot to walk might start on flat ground before introducing slopes or obstacles.
• Automatic Curriculum: Advanced methods use the agent's own learning progress to dynamically adjust the task distribution, focusing on tasks at the "edge" of its capability.

Curriculum learning acts as scaffolding, decomposing a complex target task into a sequence of simpler subtasks. This prevents the agent from being overwhelmed by the full complexity from the start, which often leads to poor exploration and reward hacking.

• Skill Chaining: A policy learns to grasp an object before learning to lift it, and then to carry it.
• State Initialization: Easier tasks often begin from "helpful" initial states (e.g., an object already near the gripper) before requiring the agent to achieve that state from a neutral start.

In sparse-reward environments, random exploration is highly inefficient. A curriculum provides a dense learning signal early on by shaping the environment or goal. As the policy improves, the curriculum gradually removes these "training wheels," returning to the original, sparse-reward problem.

• Goal Relabeling: In hindsight experience replay, failed attempts at a hard goal are relabeled as successes for easier, achieved sub-goals, creating a natural curriculum.
• Reverse Curriculum: Starting near the goal state and gradually increasing the initial distance.

Curriculum learning is most powerfully applied within Reinforcement Learning (RL) and Imitation Learning (IL) frameworks. It directly addresses key RL challenges:

• Sample Efficiency: Faster convergence by focusing on learnable tasks.
• Stability: Smoother learning curves and reduced policy collapse.
• Generalization: Exposure to a structured range of difficulties can lead to more robust policies.

Common in embodied AI and robotics, it is often combined with sim-to-real transfer and domain randomization, where the curriculum controls the realism or variability of the simulation.

Manual curriculum design is domain-specific and costly. Automatic curriculum learning algorithms design the sequence dynamically:

• Teacher-Student Frameworks: A "teacher" network (or algorithm) generates tasks for the "student" policy based on the student's performance.
• GoalGAN: Uses a Generative Adversarial Network (GAN) to generate goals of appropriate difficulty.
• Paired with Intrinsic Motivation: Algorithms like self-play or those that maximize learning progress automatically create a challenging yet surmountable sequence of tasks.

In visuomotor control, curriculum learning is essential for training policies that map pixels to actions.

• Visual Complexity: Start with simple, uncluttered backgrounds and ideal lighting, gradually introducing visual noise, distractors, and varied textures.
• Action Precision: Begin with large, coarse-grained action tolerances (e.g., reaching a region) before requiring millimeter-precision placement.
• Multimodal Integration: Early training might use perfect state information alongside images, gradually phasing out the state and forcing reliance on vision alone.

This structured approach is critical for bridging the sim-to-real gap and achieving reliable real-world robotic manipulation.

Curriculum learning is essential for training robots to perform fine-grained, contact-rich tasks like in-hand object reorientation or tool use. The curriculum typically progresses through stages:

• Stage 1: Manipulating simple, symmetric objects in a fixed hand pose.
• Stage 2: Introducing object asymmetry and mild perturbations.
• Stage 3: Requiring full re-orientation and recovery from large disturbances. This staged approach prevents the policy from becoming trapped in poor local optima associated with sparse reward signals in high-dimensional continuous action spaces.

A core application is bridging the sim2real gap for legged robots. The curriculum is defined by progressively increasing the domain randomization intensity during simulation training.

• Start with narrow parameter ranges for dynamics (friction, motor strength) and visuals (textures, lighting).
• Gradually expand these ranges to cover a vast distribution of possible real-world conditions.
• Final policy stages may include latency injection and sensor noise models. This method, used by teams like ETH Zurich's Robotic Systems Lab, produces policies robust enough to handle unseen real-world terrain like gravel, grass, and stairs.

In embodied AI tasks like Vision-and-Language Navigation (VLN), curriculum learning structures the complexity of instructions and environments.

• Easy: Short, imperative instructions ("go to the kitchen") in small, obstacle-free simulated homes.
• Medium: Longer, sequential instructions ("go past the sofa, then turn left into the bedroom") with partial clutter.
• Hard: Complex, referential instructions ("find the room with the red chair next to the window") in large, photorealistic environments with many distractors. This progression helps the agent learn to ground language in visual scenes and plan paths before tackling compositional reasoning.

Curriculum learning naturally aligns with hierarchical reinforcement learning (HRL). A high-level manager policy is trained to invoke low-level skills, which themselves are learned via curriculum.

1. Train Low-Level Primitives: Use a curriculum to learn basic skills (e.g., reach, grasp, place) in isolation.
2. Freeze Primitives: The trained skills become the action space for a higher-level policy.
3. Curriculum for the Manager: The high-level policy learns to sequence these skills, starting with short sequences (e.g., reach then grasp) and progressing to long-horizon tasks (e.g., unstack cup, place on tray, carry to table). This decomposes an otherwise intractable long-horizon problem.

In model-free reinforcement learning, curricula are designed to shape the reward function or initial state distribution to guide exploration.

• Goal Curriculum: Start training with goals that are easy to achieve (e.g., a robotic arm reaching near a target). Gradually make goals more precise (e.g., precise peg-in-hole insertion).
• Initial State Curriculum: Begin episodes from states close to success. Over time, start episodes from states farther away, forcing the policy to learn robust recovery behaviors.
• Auxiliary Reward Curriculum: Provide dense, shaped rewards initially (e.g., reward for moving closer to the goal). Phase out these shaped rewards, leaving only the final sparse success signal, to prevent reward hacking.

Curriculum learning is combined with Behavior Cloning (BC) and Inverse Reinforcement Learning (IRL) to learn from suboptimal demonstrations.

• Demonstration Difficulty: Train initially on the clearest, most successful expert demonstrations.
• Progressive Dataset Mixing: Gradually introduce noisier, more varied, or even partially failed demonstrations into the training set.
• DAgger-like Curricula: In interactive settings like Dataset Aggregation (DAgger), the curriculum can control when to query the expert—starting with simple states and progressing to more challenging, failure-prone states encountered by the learned policy. This maximizes the value of expensive expert time.

The ability of a learned policy or model to perform effectively on new, unseen data or in environments different from those encountered during training. Curriculum learning is a primary technique to improve generalization by systematically exposing the agent to a broader distribution of tasks.

• Core Goal: Prevent overfitting to narrow training conditions.
• Mechanism: The progressively harder tasks in a curriculum act as a form of data augmentation and domain expansion, forcing the policy to learn robust, transferable features.
• Outcome: A policy that can handle the sim2real gap and unexpected variations in the physical world.

A policy structured into multiple levels of abstraction, where a high-level policy selects sub-goals or skills that are executed by lower-level policies. Curriculum learning can be applied at each level.

• High-Level Curriculum: The agent first learns simple sub-goals before complex task decompositions.
• Low-Level Curriculum: A visuomotor policy might first learn coarse movements (e.g., reach) before fine dexterous manipulation.
• Synergy: This structure naturally accommodates a curriculum, as mastering low-level skills is a prerequisite for high-level planning.

A reinforcement learning paradigm where an agent learns or is given a forward dynamics model of the environment. This model can be instrumental in designing and automating a curriculum.

• Automated Curriculum: The learned model can predict task difficulty or the agent's probability of success, allowing for dynamic task sequencing.
• Safe Exploration: Training initially in the model's "mental simulation" of easy scenarios reduces risky real-world exploration.
• Planning: The model enables the agent to rehearse and plan for harder tasks within the curriculum before execution.

A technique for sim-to-real transfer where simulation parameters (e.g., textures, lighting, physics) are varied during training. It is often used in conjunction with curriculum learning.

• Curriculum of Randomization: Start with minimal randomization (a simple, consistent sim) and gradually increase the variability (e.g., add noise, change object properties).
• Objective: The policy first masters the basic task, then learns to be invariant to perceptual and dynamic nuisances.
• Result: A robust policy that bridges the sim2real gap by having experienced a vast distribution of simulated worlds.

The problem of inferring a reward function from observed optimal behavior (demonstrations). Curriculum learning can emerge from the structure of the demonstrations.

• Demonstration Curriculum: An expert naturally provides easier demonstrations before complex ones. An IRL algorithm that learns from this sequence may implicitly recover a curriculum.
• Reward Shaping: The inferred reward function can be used to define a progression of tasks, where earlier tasks have simpler reward landscapes to optimize.
• Bootstrapping: IRL can provide an initial reward for simple tasks, which is then refined as the agent progresses through harder curricula.

A policy that takes a goal specification as an additional input, enabling it to perform different tasks. This architecture is exceptionally well-suited for curriculum learning.

• Natural Curriculum: The goal space itself can be structured by difficulty (e.g., "reach near" -> "reach far" -> "pick up" -> "place precisely").
• Unified Training: A single policy can be trained across the entire curriculum by sampling goals of increasing complexity.
• Evaluation: Progress is easily measured by the policy's success rate on goals of a given difficulty tier.