Imitation Learning

The most direct approach, treating the problem as supervised learning. The agent learns a direct mapping from observed states to expert actions, minimizing the error between its predicted action and the demonstrated action.

• Input: State-action pairs from expert trajectories
• Mechanism: Minimizes a loss function like Mean Squared Error (MSE) for continuous actions or cross-entropy for discrete actions
• Key Weakness: Suffers from distributional shift; small errors compound, leading the agent to encounter states not present in the training data, causing catastrophic failure

An interactive paradigm where the agent learns by querying an expert during training. The expert provides corrective actions when the agent deviates, allowing the agent to learn a recovery policy.

• DAgger (Dataset Aggregation): The seminal DPL algorithm. The agent's policy is executed, and the expert provides ground-truth labels for the states visited by the learner, not just the expert. This data is aggregated into the training set iteratively.
• Advantage: Mitigates distributional shift by training on on-policy states
• Requirement: An interactive expert that can be queried online during the training loop

Instead of mimicking actions, IRL infers the latent reward function that the expert is implicitly optimizing. The agent then uses standard RL to learn a policy that maximizes this inferred reward.

• Core Assumption: The expert's behavior is optimal with respect to some unknown reward function R*(s, a)
• Process:
  1. Estimate R* from demonstrations
  2. Run forward RL (e.g., policy gradient) using the learned reward
• Benefit: Produces a reward function that generalizes more robustly to novel states than a cloned policy, as it captures the intent behind actions

A model-free IRL framework that frames IL as a minimax game between a generator (the agent's policy) and a discriminator. It bypasses the intermediate step of explicitly recovering a reward function.

• Discriminator: A neural network trained to distinguish between state-action pairs from the expert and the agent
• Generator (Policy): Trained via RL (e.g., TRPO, PPO) to maximize the discriminator's confusion, using the discriminator's output as a cost signal
• Outcome: The agent's state-action distribution is directly matched to the expert's, achieving high-fidelity behavior cloning without per-step supervision

A foundational IRL approach that assumes the true reward function is a linear combination of known features. The goal is to find a policy whose expected feature counts match those of the expert.

• Feature Expectation: The average value of a feature vector φ(s) accumulated over a trajectory
• Algorithm: Iteratively solves a quadratic programming problem to find a reward weight vector that maximizes the margin between the expert's feature expectations and all previously generated policies
• Guarantee: Provides a theoretical bound on the policy's performance relative to the expert, assuming the linear reward structure holds

Extreme IL settings where the agent must learn a new task from a single demonstration or a very small number of them. This requires strong priors or meta-learning.

• Goal-Conditioned BC: The policy is conditioned on a goal state, allowing it to generalize a demonstrated sequence to a new target
• Meta-IL (MAML): Uses Model-Agnostic Meta-Learning to train an initialization policy that can adapt to a new task with a single gradient step on the demonstration data
• Key Enabler: Leverages pre-training on a wide distribution of related tasks to build a reusable behavioral prior

The simplest form of imitation learning that treats policy learning as a supervised learning problem over state-action pairs from expert trajectories.

• Directly maps observations to actions using a dataset of expert demonstrations
• Minimizes the error between the agent's action and the expert's action in any given state
• Key Limitation: Suffers from covariate shift—small compounding errors cause the agent to drift into states not seen in the training data, leading to catastrophic failure

A paradigm where the agent infers the underlying reward function that the expert is implicitly optimizing, rather than directly copying actions.

• Given a set of expert trajectories, IRL extracts a reward signal that explains the observed behavior
• The inferred reward is then used with standard reinforcement learning to derive a policy
• Advantage: Generalizes better to unseen states than behavioral cloning because it captures the expert's intent, not just their actions

A model-free imitation learning framework that frames the problem as a minimax game between a generator policy and a discriminator.

• The discriminator learns to distinguish between expert state-action pairs and those generated by the agent's policy
• The generator (agent policy) is trained via reinforcement learning to fool the discriminator
• Eliminates the need to manually specify or infer a reward function; the discriminator's confusion serves as the reward signal

An iterative algorithm designed to mitigate the covariate shift problem inherent in standard behavioral cloning.

• The agent's current policy is executed to collect on-policy trajectories
• A human expert provides corrective labels for the states the agent actually visits
• These new labeled states are aggregated into the original training dataset
• The policy is retrained on this augmented dataset, and the process repeats, ensuring the model learns to recover from its own mistakes

A formal framework where an agent learns to perform a task by observing an expert, with the goal of achieving performance comparable to the expert.

• Assumes the expert's policy is optimal with respect to an unknown reward function that is a linear combination of known features
• The algorithm iteratively finds a reward function that maximally separates the expert's feature expectations from the agent's current policy
• Guarantees that the learned policy's performance converges to within a bounded margin of the expert's performance

A learning paradigm where an agent must learn an optimal policy entirely from a fixed, static dataset of previously collected interactions, without any further online exploration.

• Directly relevant to imitation learning when expert data is treated as the offline dataset
• Key Challenge: The distributional shift between the learned policy's actions and the actions in the dataset leads to overestimated Q-values for out-of-distribution actions
• Techniques like Conservative Q-Learning (CQL) penalize Q-values for unseen actions to stay within the data's support