Inverse Reinforcement Learning (IRL)

IRL addresses the fundamental challenge of reward specification. Rather than hand-engineering a reward function—which often leads to reward hacking or unintended behaviors—IRL observes expert trajectories and deduces the underlying objective. The core assumption is that the expert's demonstrated behavior is optimal with respect to some unknown reward function. The algorithm searches for a reward function for which the observed expert policy is uniquely optimal, effectively extracting the expert's implicit intent from behavioral data.

A foundational framework that resolves the ambiguity problem inherent in IRL, where multiple reward functions can explain the same expert behavior. Maximum Entropy IRL selects the reward function that makes the expert's demonstrated trajectories maximally likely while otherwise maximizing entropy over all possible paths. This principle:

• Prevents the model from assuming arbitrary preferences in unobserved regions of the state space
• Produces a stochastic policy that matches expert feature expectations
• Provides a probabilistic generative model of expert behavior
• Forms the theoretical basis for modern Generative Adversarial Imitation Learning (GAIL)

A core algorithmic approach where IRL finds a reward function such that the expected feature counts of the learned policy match those of the expert demonstrations. The process iterates between:

• Estimating the expert's feature expectations from observed trajectories
• Computing the learned policy's feature expectations under the current reward hypothesis
• Adjusting the reward weights to minimize the discrepancy This method ensures the agent's long-run behavior statistically aligns with the expert's demonstrated preferences without requiring explicit action-by-action mimicry.

A practical framework where IRL is used as an intermediate step to learn a policy that performs as well as the expert, rather than recovering the exact reward function. The algorithm maintains a convex set of candidate reward functions consistent with expert demonstrations and selects the one that maximizes the margin between the expert's policy and the current learned policy. Key properties:

• Provides theoretical guarantees on policy performance bounds relative to the expert
• Requires only trajectory data, not explicit action labels
• Generalizes to states not visited by the expert
• Forms the bridge between pure IRL and direct policy imitation

A probabilistic formulation that treats the reward function as a latent random variable and computes a posterior distribution over possible reward functions given the expert demonstrations. This approach:

• Quantifies uncertainty about the true reward function
• Uses a prior distribution to encode assumptions about reward structure
• Enables active learning by identifying states where the reward posterior has high entropy
• Allows the agent to reason about multiple plausible explanations for expert behavior
• Naturally handles suboptimal or noisy expert demonstrations by modeling them as evidence rather than absolute truth

While Behavioral Cloning directly learns a mapping from states to expert actions via supervised learning, IRL infers the underlying reward function first. This distinction creates critical differences:

• Distributional Shift: Behavioral cloning suffers from compounding errors when the agent deviates from expert trajectories; IRL learns a reward that guides recovery from novel states
• Generalization: IRL can discover policies that outperform the expert if the inferred reward is more accurate than the expert's own policy execution
• Data Efficiency: IRL requires fewer demonstrations to generalize to new environments because it captures intent, not just mimicry
• Interpretability: The recovered reward function provides a human-readable explanation of expert preferences

The broader training paradigm where an agent learns a policy by observing expert demonstrations rather than through trial-and-error or a pre-defined reward function. Behavioral Cloning (BC) directly maps states to actions via supervised learning but suffers from compounding error due to covariate shift. IRL is a distinct branch of imitation learning that first recovers the expert's latent objective before deriving a policy, offering superior generalization to unseen states.

The mathematical framework for modeling sequential decision-making in stochastic environments, defined by the tuple (S, A, P, R, γ). In standard RL, the reward function R is known. IRL inverts this: given an optimal or near-optimal policy π* in an MDP\R (an MDP without a reward function), the goal is to recover the R that explains the observed behavior. The assumption is that the expert is optimizing some unknown cumulative reward.

A specific IRL framework introduced by Abbeel and Ng (2004) that assumes the true reward function is a linear combination of known features. Instead of recovering the exact reward, it finds a policy whose feature expectations match those of the expert's demonstrated trajectories. The algorithm iteratively inverts the RL problem to find a reward function that maximally penalizes the current policy's deviation from expert feature counts.

A probabilistic framework introduced by Ziebart et al. (2008) that resolves the reward ambiguity problem inherent in IRL. It models expert behavior as a Boltzmann distribution over trajectories, where the probability of a path is proportional to the exponential of its cumulative reward. This yields a unique, principled solution by assuming the expert is maximizing reward while also acting as randomly as possible, matching feature counts without overcommitting to a single policy.

A model-free imitation learning algorithm that frames the problem as a minimax game between a generator (the policy) and a discriminator. The discriminator is trained to distinguish between expert state-action pairs and those generated by the policy, while the generator is optimized to fool the discriminator. GAIL implicitly performs IRL by using the discriminator's output as a learned, non-stationary cost function, bypassing explicit reward function recovery.

The fundamental ill-posed nature of IRL: infinitely many reward functions can explain a given optimal policy. For example, a zero reward everywhere makes any policy optimal. Effective IRL algorithms impose structural assumptions or regularization to select a unique solution. Common resolutions include maximum margin principles, maximum entropy, or assuming the reward is a linear combination of sparse, known features.