Inverse Reinforcement Learning (IRL)

Standard RL solves for an optimal policy given a reward function. IRL solves the inverse problem: given an optimal or near-optimal policy (demonstrated via trajectories), infer the reward function being maximized. This is an ill-posed problem because many reward functions can explain the same observed behavior, requiring additional constraints like the maximum entropy principle to select a unique solution.

A foundational framework that resolves the ambiguity of the inverse problem by assuming the expert follows a probabilistic policy that maximizes reward while also maximizing entropy. This yields a Boltzmann-rational policy where the probability of an action is proportional to the exponential of its expected Q-value. The approach recovers a reward function that explains behavior without assuming absolute optimality, making it robust to suboptimal demonstrations.

A core constraint in IRL algorithms: the expected feature counts of the learned policy must match those of the expert's demonstrated trajectories. This ensures the agent's behavior aligns with the expert's visitation patterns in state space. Algorithms like Apprenticeship Learning via Inverse Reinforcement Learning use this principle to find a reward function that makes the expert's feature expectations outperform all other policies by a margin.

A modern IRL approach that frames the problem as a minimax game between a generator (policy) and a discriminator. The discriminator learns to distinguish between expert and generated state-action pairs, effectively recovering a reward function without explicitly computing it. The generator uses Trust Region Policy Optimization (TRPO) to fool the discriminator. GAIL is model-free and scales to high-dimensional continuous control tasks.

A probabilistic framework that treats the reward function as a latent variable and computes a posterior distribution over possible reward functions given the expert's demonstrations. This quantifies epistemic uncertainty about the true reward and allows for active learning by selecting queries that maximize information gain. Bayesian IRL uses Markov Chain Monte Carlo (MCMC) or variational inference to sample from the posterior.

While reward shaping manually adds heuristic bonuses to guide an agent, IRL learns the reward from data. IRL is preferred when the desired behavior is easier to demonstrate than to encode as a reward function—such as teaching a robot to pour coffee or drive defensively. The inferred reward often transfers better to new environments than a hand-shaped reward because it captures the underlying intent rather than surface-level heuristics.

The process of breaking down a scalar reward signal into constituent sub-rewards to explain which objectives are driving an agent's behavior. While IRL infers a reward function from demonstrations, reward decomposition disentangles an already-known reward into semantically meaningful components.

• Additive decomposition: $R(s,a) = w_1R_1(s,a) + w_2R_2(s,a)$
• Enables answering: 'Did the agent brake for safety or for comfort?'
• Complements IRL by validating inferred rewards against human priors

A technique for rendering an agent's learned policy as a visual heatmap or graph to illustrate which actions it will take in specific states. When combined with IRL, the inferred reward function can be used to generate counterfactual policy visualizations.

• State-value heatmaps show desirability of regions
• Action arrows overlay preferred trajectories
• Critical for validating that an inferred reward produces behavior aligned with expert intent

The foundational mathematical framework for reinforcement learning, defined by states, actions, a transition function, and a reward function. IRL operates on the assumption that expert demonstrations were generated by an agent optimizing an unknown reward within an MDP.

• Tuple: $(S, A, T, R, \gamma)$
• IRL inverts the standard RL problem: given optimal behavior, find $R$
• The quality of inferred rewards depends on MDP specification fidelity

The application of causal inference tools, like intervention analysis, to determine whether a policy relies on spurious correlations or true causal relationships. IRL-inferred rewards can encode spurious correlations present in expert demonstrations.

• Do-calculus tests if changing a feature changes the policy
• Identifies whether an inferred reward captures causation or mere association
• Essential for safety-critical deployment of IRL-derived policies

An explanation format that answers 'Why action A instead of action B?' by highlighting the minimal state differences that caused the policy to diverge. In IRL contexts, contrastive explanations reveal which features of the inferred reward function drive decision boundaries.

• Generates minimal counterfactual states
• Exposes the implicit preferences encoded in the learned reward
• Bridges the gap between a numeric reward vector and human-understandable rationale

A causal interpretability method that systematically removes or occludes input features to measure the resulting change in policy output and determine feature necessity. Applied to IRL, ablation validates whether the inferred reward function depends on semantically meaningful state dimensions.

• Zero-out or permute individual state features
• Measure drop in explained variance of expert behavior
• Identifies which features the expert is inferred to care about most