Imitation Learning

The foundational IL technique that treats policy learning as a supervised learning problem. The agent learns a direct mapping from observed states to expert actions by minimizing the error between its predicted action and the demonstrated action.

• Input: State-action pairs from expert trajectories
• Mechanism: Minimizes a loss function (e.g., mean squared error for continuous actions, cross-entropy for discrete)
• Key Limitation: Suffers from covariate shift—small errors compound over time, leading the agent into states not seen in the training distribution

Instead of directly copying actions, IRL infers the underlying reward function that the expert is implicitly optimizing. The agent then uses standard RL to learn a policy from this recovered reward.

• Process: Observe expert → Recover reward function → Train policy via RL
• Advantage: Generalizes better than BC because the agent understands why an action was taken, not just what action was taken
• Algorithms: Maximum Entropy IRL, Maximum Margin Planning, Bayesian IRL

A model-free IL framework that frames learning as a minimax game between a generator (the policy) and a discriminator. The discriminator learns to distinguish between expert and agent state-action pairs, while the policy learns to fool the discriminator.

• Architecture: Inspired by Generative Adversarial Networks (GANs)
• Key Benefit: Avoids the intermediate step of reward function recovery required by IRL
• Sample Efficiency: Requires significant environment interactions compared to pure BC but achieves more robust policies

An iterative algorithm designed to mitigate the covariate shift problem inherent in standard Behavioral Cloning. DAgger actively queries the expert for corrective labels on states visited by the learner's current policy.

• Loop: Train policy → Execute policy → Expert labels visited states → Aggregate into dataset → Retrain
• Result: The training distribution converges to the policy's induced state distribution, not just the expert's
• Constraint: Requires an interactive expert available for online labeling during training

Advanced IL techniques that enable an agent to learn a new task from a single demonstration or a very small number of examples. This is achieved through meta-learning or learning invariant task representations.

• Goal: Generalize to novel tasks without retraining from scratch
• Mechanism: Often uses contextual policies conditioned on the demonstration itself
• Application: Critical for warehouse robots that must quickly adapt to new pick-and-place operations without extensive reprogramming

A paradigm where the agent learns exclusively from a static, pre-collected dataset of expert demonstrations without any further interaction with the environment or expert during training.

• Constraint: No online exploration or expert queries allowed
• Challenge: Distributional shift is severe; the agent cannot correct its mistakes by asking the expert
• Techniques: Employs conservative policy updates and value regularization to avoid overestimating out-of-distribution actions

The most direct form of imitation learning, where the agent's policy is trained via supervised learning to map observed states directly to the expert's actions. The training dataset consists of state-action pairs from demonstrations.

• Key Limitation: Suffers from covariate shift—small errors compound over time, leading the agent into states not seen in the training data.
• Common Use: Initializing a policy before fine-tuning with reinforcement learning.
• Architecture: Often implemented as a deep neural network minimizing the mean squared error or cross-entropy loss between predicted and expert actions.

Instead of copying actions, IRL infers the underlying reward function that the expert is implicitly optimizing. The agent then uses this learned reward to train its own policy via standard RL.

• Core Insight: The expert's behavior is treated as the optimal solution to an unknown MDP.
• Advantage: Produces policies that generalize better to novel states than behavioral cloning because they understand the why behind actions.
• Algorithms: Maximum Entropy IRL, Maximum Margin Planning.

A model-free imitation learning framework that frames the problem as a minimax game between a generator (the agent's policy) and a discriminator.

• Mechanism: The discriminator is trained to distinguish between expert state-action pairs and agent-generated pairs. The generator is trained via TRPO or PPO to fool the discriminator.
• Key Benefit: Avoids the intermediate step of reward function inference required by IRL, directly recovering a policy that is indistinguishable from the expert.
• Connection: Draws a direct analogy to Generative Adversarial Networks (GANs) in computer vision.

An iterative algorithm designed to mitigate the distributional shift problem inherent in standard behavioral cloning.

• Process: The agent executes its current policy, and a human expert provides corrective labels for the states the agent actually visits. This new data is aggregated into the training set.
• Loop: Train policy → Execute policy → Collect expert corrections on visited states → Aggregate data → Retrain.
• Result: The policy is trained on on-policy data, ensuring it learns to recover from its own mistakes rather than just mimicking perfect trajectories.

A closely related paradigm where an agent learns a policy exclusively from a static, pre-collected dataset of transitions, without any online interaction with the environment.

• Relationship: Imitation learning can be viewed as a subset of offline RL where the dataset consists only of expert demonstrations.
• Core Challenge: Extrapolation error—the agent overestimates the value of actions not present in the dataset.
• Solutions: Conservative Q-Learning (CQL) and implicit Q-learning constrain the learned policy to stay within the support of the data distribution.

A formal framework that combines IRL with RL to learn from an expert. The goal is to find a policy whose feature expectations match those of the expert's demonstrated trajectories.

• Mathematical Basis: Formulated by Abbeel and Ng (2004), it guarantees that the learned policy performs at least as well as the expert on the unknown true reward function.
• Algorithm: Iteratively projects the agent's feature expectations onto the convex hull of known policies to minimize the margin.
• Significance: Provided the first strong theoretical guarantees for learning from demonstration without access to a reward signal.