Curriculum Learning

The core mechanism involves sequencing tasks from simple to complex. This is often formalized by a task scheduler that defines a curriculum, a sequence of tasks $\mathcal{T}_1, \mathcal{T}_2, ..., \mathcal{T}_n$, where complexity increases. The agent masters foundational skills (e.g., balancing) before advancing to composite skills (e.g., walking then running). This reduces the exploration space initially, providing a stronger learning signal and preventing the agent from being overwhelmed by the full problem complexity from the start.

Manually designing a curriculum is often infeasible. Automatic curriculum learning algorithms dynamically generate the task sequence. Key methods include:

• Goal-Based Curricula: The agent proposes its own goals, with the curriculum defined by the goal achievement rate. Easier goals are sampled more frequently initially.
• Self-Paced Learning: The agent itself gauges its competence on tasks, effectively deciding when to progress.
• Teacher-Student Frameworks: A separate teacher model learns to propose tasks that maximize the learning progress of the student agent, often using metrics like the gradient norm or value function loss.

Curriculum Learning is a critical enabler for Sim-to-Real Transfer. The curriculum can be designed across simulation parameters, not just task objectives. For example:

• Start training in a low-fidelity simulation with simplified physics (e.g., no friction, perfect actuators).
• Gradually introduce domain randomization parameters (e.g., varying masses, friction coefficients, visual textures).
• This creates a smooth pathway from an easy, forgiving simulation to a highly randomized one that better covers the reality gap, resulting in a more robust policy for physical deployment.

In sparse reward environments common in robotics, random exploration rarely stumbles upon success. A curriculum structures exploration by providing intermediate reward signals or shaped rewards for simpler sub-tasks. By first learning in a dense-reward version of the task (e.g., reward for moving closer to a target), the agent acquires useful behaviors. The curriculum then anneals this shaping, guiding the agent toward solving the original sparse-reward problem (e.g., reward only for reaching the target). This directly addresses the exploration-exploitation tradeoff in early training.

By avoiding initial training on intractable tasks, Curriculum Learning significantly improves sample efficiency. The agent spends less time in unproductive regions of the state-action space. Empirical results in domains like robot manipulation and locomotion consistently show that curriculum-trained policies:

• Converge to a higher asymptotic performance.
• Require fewer total environment interactions (samples) to reach a performance threshold.
• Exhibit more stable and monotonic learning curves compared to training on the final task from scratch.

Curriculum Learning is closely related to Transfer Learning and Multi-Task Learning. It can be viewed as sequential transfer, where knowledge from earlier tasks is transferred to later, more difficult ones. The curriculum defines a task embedding space where proximity indicates transferability. This contrasts with parallel multi-task learning, where all tasks are learned simultaneously. The curriculum ensures positive forward transfer while minimizing catastrophic forgetting of earlier skills, as they remain relevant for later stages of the curriculum.

A specific neural architecture designed for transfer learning and continual learning. It trains a sequence of networks where each new network (or "column") is conditioned on the outputs of previously trained, frozen networks. This prevents catastrophic forgetting and enables the progressive acquisition of complex skills, making it a powerful architectural instantiation of curriculum learning principles for sequential task learning.

A variant of curriculum learning where the difficulty schedule is determined automatically by the learning algorithm itself, not pre-defined by a human. The model evaluates its own performance on data samples and selects easier examples first, gradually increasing difficulty as its confidence grows. This creates a data-driven curriculum that adapts to the learner's pace.

• Core Mechanism: A loss function incorporates a self-paced regularizer that weights training samples.
• Key Benefit: Eliminates the need for manual task ordering, enabling fully automated curriculum generation.

A multi-agent framework where a teacher model (or algorithm) designs the curriculum for a student model. The teacher's goal is to generate or select training tasks that maximize the student's learning progress. This is often framed as a meta-optimization or reinforcement learning problem where the teacher's actions are task proposals and its reward is the student's improvement.

• Application: Used in complex multi-agent reinforcement learning and robotics where designing a manual curriculum is intractable.
• Outcome: Can discover non-intuitive, highly effective training sequences that outperform human-designed curricula.

A reinforcement learning paradigm where the policy and value functions are conditioned on a goal state. This framework naturally lends itself to curriculum learning by sequentially increasing the difficulty of the goals presented to the agent. Training might start with goals very close to the start state and progressively expand to distant, complex goal configurations.

• Synergy with Curriculum Learning: Provides a clean, parameterized way to define task difficulty (e.g., Euclidean distance to goal).
• Example: Training a robotic arm to reach any point in its workspace, beginning with easy, reachable points.

The process of automatically creating a distribution of tasks for training, often used in conjunction with curriculum learning. Instead of manually defining each training scenario, algorithms procedurally generate tasks with tunable parameters that control difficulty (e.g., obstacle density, object mass, terrain roughness).

• Key for Sim-to-Real: Enables massive domain randomization by generating a vast, diverse set of simulation environments.
• Curriculum Integration: The task generator's parameters are adjusted over time to produce progressively harder instances, forming an automated curriculum.

Metrics used to automatically schedule a curriculum by measuring how quickly a model is learning. Instead of using pre-defined difficulty, the system prioritizes tasks where the agent is experiencing high learning progress—the derivative of performance with respect to training time or samples.

• Mechanism: The agent or teacher model estimates its own improvement rate on different task families.
• Outcome: Focuses training on tasks at the "zone of proximal development" where learning is fastest, avoiding tasks that are too easy (no progress) or too hard (plateau). This is a core component of advanced self-paced and teacher-student methods.