Expert Policy

The core assumption in imitation learning is that the expert policy represents a high-performing, often near-optimal, strategy for the task. This does not require the policy to be perfect, but it must be sufficiently good that mimicking its behavior leads to successful task completion. The strength of this assumption varies:

• Strict Optimality: The policy is the mathematically optimal solution to a known Markov Decision Process (MDP). This is rare in practice.
• Practical Optimality: The policy performs the task reliably and efficiently, as judged by a human domain expert (e.g., a skilled surgeon or factory worker).
• Suboptimal Demonstrations: Many real-world datasets contain noise, mistakes, or stylistic variations. Algorithms must be robust to this demonstration noise.

Typically, the internal mechanics of the expert policy are not accessible to the imitation learning algorithm. The learner only observes the policy's outputs—demonstration trajectories—not its parameters, architecture, or internal reasoning. This black-box constraint shapes the field:

• Behavioral Cloning treats the policy as a function mapping states to actions and learns a direct approximation.
• Inverse Reinforcement Learning (IRL) attempts to infer the reward function that the expert is implicitly optimizing, a more general but ill-posed problem.
• The expert can be a human, a pre-programmed controller, or another trained AI agent. The learning challenge increases with the expert's complexity and the noisiness of the observations.

The expert policy induces a unique probability distribution over state-action pairs, known as its occupancy measure. A primary goal of many imitation learning algorithms is to match the learner's occupancy measure to the expert's. This distributional view is powerful because:

• It moves beyond copying individual actions to replicating the long-term behavior distribution.
• Adversarial Imitation Learning methods like GAIL use a discriminator to directly match these distributions.
• The distribution defines what states are "relevant" for the task. A key challenge is covariate shift, where the learner drifts into states not covered by the expert's distribution, leading to compounding errors.

The form of the expert's output dictates the imitation learning paradigm. The policy itself is abstract, but its demonstrations are concrete data.

• State-Action Pairs: The most direct form, providing both the observed situation and the expert's chosen action. Used in Behavioral Cloning.
• State-Only Trajectories: The learner observes the sequence of states (e.g., video of a task) but not the underlying actions. This defines Imitation Learning from Observations (IfO), requiring the algorithm to infer actions.
• Third-Person Demonstrations: Observations are from a different viewpoint than the agent's own sensors (e.g., a robot learning by watching a human). This requires cross-viewpoint representation learning.
• Teleoperation & Kinesthetic Teaching: Common methods for physically collecting demonstrations from a human expert guiding a robot.

The expert policy is not just data; it defines the learning objective. Different algorithms interact with the expert policy in distinct ways:

• Supervised Learning (Behavioral Cloning): The expert policy provides labeled training data (state, action).
• Distribution Matching (GAIL, ValueDICE): The expert policy defines a target distribution that the learner's policy must mimic.
• Reward Inference (IRL): The expert policy is assumed to be optimal with respect to an unknown reward function, which the algorithm must recover.
• Interactive Learning (DAgger): The expert policy is queried online to provide corrective labels on states visited by the learner, mitigating covariate shift.

The idealized concept of an expert policy confronts practical realities that influence system design.

• Suboptimality & Noise: Real demonstrations are rarely perfect. Algorithms must filter noise or learn a robust policy that outperforms the average demonstration.
• Coverage: The expert's demonstrations provide a finite sample of its state-action distribution. The learner must generalize to unseen states, a challenge addressed by algorithms like DAgger.
• Non-Markovian Behavior: Human experts often use memory and context not captured in the immediate state. This can lead to inconsistencies if the learner's policy is Markovian.
• Reward Ambiguity: In IRL, infinitely many reward functions can explain a finite set of demonstrations. Maximum Entropy IRL resolves this by choosing the least committed, most uncertain reward function consistent with the data.

A supervised learning approach where a policy is trained to directly map observed states to expert actions. It minimizes prediction error on a static dataset but is susceptible to compounding errors due to covariate shift when the learner's state distribution drifts from the expert's.

• Core Mechanism: Treats imitation as a standard regression or classification problem on state-action pairs.
• Primary Limitation: Lacks a mechanism for recovery from errors, as it never learns what to do in states not visited by the expert.

The algorithmic framework for inferring the underlying reward function that explains the expert's behavior, based on the principle that demonstrations are optimal with respect to some unknown reward. It resolves the reward ambiguity problem by typically assuming a maximum entropy distribution over expert trajectories.

• Key Output: A learned reward function, not just a policy.
• Process: The algorithm posits a reward function, computes the optimal policy for it, and adjusts the reward to make the expert's trajectories appear optimal.

An adversarial imitation learning framework that directly learns a policy without recovering a reward function. It trains a policy (generator) to produce trajectories that a discriminator cannot distinguish from expert demonstrations.

• Analogy: Functions similarly to a GAN, but for matching state-action distributions.
• Advantage: Often more sample-efficient than pure IRL and more robust than behavioral cloning, as it actively matches distributions.

An iterative, interactive algorithm designed to combat covariate shift in behavioral cloning. It collects corrective actions from the expert on states visited by the learner's current policy, aggregating them into an expanding training dataset.

• Core Loop: 1) Train policy on current dataset. 2) Roll out policy. 3) Query expert for correct actions on visited states. 4) Aggregate new data.
• Outcome: The final dataset better covers the state distribution induced by the learned policy, reducing compounding errors.

The fundamental distributional problem in imitation learning where the state visitation distribution of the learning agent diverges from that of the expert. This mismatch occurs because early errors by the learner lead it into states not present in the original demonstration data.

• Consequence: The policy is asked to make predictions on out-of-distribution (OOD) states, where its behavior is undefined and often erroneous.
• Solution Target: Algorithms like DAgger and adversarial methods (GAIL) explicitly aim to minimize or correct for this shift.

The primary data source for imitation learning: a time-ordered sequence recorded while an expert performs a task. It can be a sequence of state-action pairs or, in the case of Imitation from Observations (IfO), states only.

• Common Formats: τ = [(s₀, a₀), (s₁, a₁), ..., (s_T, a_T)]
• Collection Methods: Includes kinesthetic teaching (physically guiding a robot), teleoperation, motion capture, or human video recordings.
• Data Quality: The assumption of optimality is often relaxed in practice, leading to work on learning from suboptimal demonstrations.