Curriculum Learning

The core mechanism involves sequencing tasks from simple to complex. This is analogous to human education, where foundational concepts are mastered before advanced topics. In RL, this often means:

• Starting in toy environments with sparse rewards or simple dynamics.
• Gradually introducing complex state spaces, longer horizons, or sparse/delayed rewards.
• The progression can be defined by environment parameters (e.g., obstacle density, action noise) or task definitions (e.g., reach a nearby goal before a distant one).

Manually designing a curriculum is labor-intensive. Modern approaches use automated curriculum learning where the agent or a teacher algorithm generates its own syllabus. Key methods include:

• Self-Paced Learning: The agent samples tasks based on its current performance, favoring those where it is neither failing completely nor succeeding perfectly.
• Goal-Conditioned RL: The agent learns to achieve any goal in a space; the curriculum is defined by progressively sampling more challenging target goals.
• Teacher-Student Frameworks: A separate teacher network is trained to propose tasks that maximize the learning progress of the student agent, often using concepts like learning progress or prediction error as a signal.

Curriculum learning directly addresses core RL challenges:

• Sample Efficiency: By mastering simple tasks first, the agent learns useful skills and representations (e.g., navigation, object interaction) that transfer to harder tasks, reducing the total number of environment interactions needed.
• Training Stability: It mitigates the exploration problem in sparse-reward settings. Early success in easy tasks provides a reward signal that guides exploration in harder settings, preventing the agent from getting stuck with zero reward.
• It helps avoid catastrophic forgetting of early skills by continuously practicing them as part of more complex task sequences.

Curriculum learning is intrinsically linked to synthetic data generation and simulated environments:

• Simulation Enables Curriculum: High-fidelity physics engines and procedural generation allow for the creation of infinite, parameterized task variations essential for building a curriculum.
• Domain Randomization as a Curriculum: Varying simulation parameters (e.g., friction, lighting) can be scheduled from a narrow, easy distribution to a broad, realistic one, acting as a curriculum for sim-to-real transfer.
• World Models: An agent can train on a curriculum of imagined trajectories within a learned world model, safely exploring dangerous or costly state spaces in a synthetic latent environment.

From a theoretical perspective, curriculum learning frames training within a Non-Stationary Markov Decision Process. The environment's transition function T(s'|s,a) or reward function R(s,a) changes over time according to a curriculum schedule.

• The agent must adapt to this shifting distribution of tasks.
• The optimal schedule is one that maximizes the agent's final performance on a target task distribution, not its performance on the intermediate tasks.
• This formalization connects to meta-learning and continual learning, where the ability to adapt quickly is key.

The success of a curriculum is measured by comparing an agent's learning curve to baselines:

• Final Asymptotic Performance: Does the curriculum-trained agent achieve a higher final reward on the target task?
• Learning Speed (Wall-clock Time): How many environment steps or training iterations are required to reach a performance threshold?
• Sample Efficiency: The total number of experiences needed.
• Robustness & Generalization: Performance on a held-out set of test tasks or in the real world after sim-to-real transfer.
• A failed curriculum can lead to negative transfer or catastrophic forgetting if the progression is too abrupt.

A computational model of a real or abstract world, defined by state/action spaces, transition dynamics, and a reward function. It is the foundational synthetic training ground where curriculum learning is typically applied. Without a high-fidelity simulation, creating a structured curriculum of tasks is impractical.

• Core Components: State space, action space, transition function (dynamics), reward function.
• Primary Use: Provides a safe, scalable, and controllable domain for RL training, enabling the precise difficulty scaling required for curriculum learning.

An algorithmic method for creating data—such as environments, levels, or tasks—with near-infinite variation. It is a key enabler for curriculum learning by automatically generating a vast, diverse set of training scenarios of controllable difficulty.

• Mechanism: Uses rules, noise functions, or learned models to create new content.
• Application in Curriculum Learning: Allows for the automatic creation of the next task in a curriculum by parameterizing difficulty (e.g., maze complexity, number of enemies, physical parameters).

A drive for an agent to explore based on internally generated rewards, such as curiosity or novelty. It is often used as an automatic mechanism for curriculum generation, where the agent seeks out states or tasks that maximize learning progress.

• Common Forms: Prediction error (curiosity), state visitation counts (novelty), learning progress.
• Relation to Curriculum: Can replace a hand-designed curriculum by having the agent self-select tasks that are neither too easy (boring) nor too hard (unlearnable), effectively creating its own adaptive curriculum.

A framework where an agent operates at multiple levels of temporal abstraction, using high-level skills (options) over extended time horizons. Curriculum learning can be applied hierarchically, where the agent first masters low-level skills before sequencing them into complex high-level behaviors.

• Structure: A high-level manager policy sets goals for a low-level worker policy.
• Synergy with Curriculum: The curriculum can first train the low-level skills in isolation (e.g., walking, grasping) before training the manager to compose them for long-horizon tasks (e.g., navigate to object and pick it up).

A sim-to-real transfer technique where parameters of a simulated environment (e.g., textures, lighting, physics) are varied during training. It can be integrated with curriculum learning by starting with a narrow, easy distribution of parameters and gradually expanding it to a broad, realistic one.

• Goal: Learn a policy robust to the variability of the real world.
• Curriculum Integration: The "difficulty" in the curriculum is defined by the breadth and realism of the randomized parameters, smoothly bridging the reality gap.

A class of algorithms where an agent learns or uses a model of the environment's dynamics. This learned model can be leveraged to generate synthetic experience or "imagine" rollouts, which is powerful for curriculum learning. The agent can practice in its own mental model of increasingly challenging scenarios before facing the real environment.

• Key Component: The learned transition and reward models.
• Curriculum Application: The agent can use its world model to generate and practice on predicted future difficult states, creating a self-supervised curriculum in a latent space.