Inverse Reinforcement Learning (IRL)

IRL solves the problem of reward function ambiguity. Given a set of expert trajectories, the goal is to find a reward function under which the expert's behavior is uniquely optimal.

• Forward RL: Reward → Policy
• Inverse RL: Policy (Demonstrations) → Reward
• The inferred reward explains why an expert took specific actions, not just what actions they took.
• This is an ill-posed problem because many reward functions can explain the same observed behavior.

A foundational framework that resolves reward ambiguity by assuming the expert follows a maximum entropy policy—one that achieves high reward while acting as randomly as possible within constraints.

• Matches feature expectations between the learned policy and expert demonstrations.
• Produces a probabilistic model of behavior, assigning higher likelihood to trajectories with higher cumulative reward.
• The resulting reward function is the one that maximizes the log-likelihood of observed expert trajectories.
• Forms the theoretical basis for modern Generative Adversarial Imitation Learning (GAIL).

A practical framework where an agent learns to perform a task by observing expert demonstrations, without ever receiving explicit rewards.

• Step 1: Estimate the expert's reward function from demonstrations using IRL.
• Step 2: Use standard RL to learn a policy that optimizes the inferred reward.
• The learned policy should achieve performance comparable to the expert, even in states not visited during demonstration.
• This approach generalizes beyond mimicry—the agent can recover from errors the expert never made.

Treats the reward function as a random variable and maintains a posterior distribution over possible reward functions given observed demonstrations.

• Prior distribution encodes assumptions about reward structure (e.g., sparsity, smoothness).
• Likelihood models the probability of expert actions given a candidate reward function, typically using a Boltzmann rationality model.
• Posterior inference yields a distribution over reward functions, capturing uncertainty about the expert's true intent.
• Enables active learning: the agent can query the expert in states where reward uncertainty is highest.

IRL is uniquely suited for quantitative finance because human traders rarely articulate their exact reward function, yet produce rich demonstration data through order flow.

• Infer risk preferences: Recover a trader's implicit risk-aversion coefficient from their execution patterns.
• Model market maker intent: Deduce the utility function driving spread placement and inventory management.
• Detect strategy drift: Monitor whether a trader's inferred reward function shifts over time, signaling behavioral change.
• The inferred reward can serve as a portable model of expertise, transferable across similar market conditions.

A core mathematical condition in many IRL algorithms requiring that the expected feature counts of the learned policy match those of the expert's demonstrated trajectories.

• Features are hand-crafted functions mapping state-action pairs to scalar values (e.g., inventory level, spread capture).
• The reward function is assumed to be a linear combination of these features: R(s,a) = ωᵀφ(s,a).
• Matching feature expectations ensures the learned policy visits similar state distributions as the expert.
• This constraint alone is insufficient for unique reward recovery; additional criteria like maximum entropy are required.

The mathematical bedrock upon which IRL is built. An MDP formalizes sequential decision-making using states, actions, transition probabilities, and a reward function. In IRL, the reward function is the unknown variable to be inferred, while the expert's demonstrated state-action trajectories provide the observed data. The goal is to recover the reward structure that makes the expert's behavior optimal under the MDP framework.

A probabilistic extension that resolves the ambiguity problem in standard IRL—where many reward functions can explain the same expert behavior. MaxEnt IRL assumes the expert follows a stochastic policy that maximizes entropy subject to feature matching constraints. The resulting model assigns higher probability to trajectories with higher cumulative reward, producing a distribution over demonstrated paths rather than a single deterministic policy. This is critical for capturing the inherent stochasticity in human trading decisions.

A complementary technique to IRL where domain knowledge is used to engineer auxiliary reward signals that accelerate learning. While IRL infers the reward function from data, reward shaping manually injects heuristic guidance. The two approaches can be combined: IRL recovers the core objective from expert demonstrations, while potential-based reward shaping provides dense intermediate feedback to speed up convergence in sparse-reward trading environments.

The simplest form of imitation learning that treats the problem as supervised learning—directly mapping observed states to expert actions. Unlike IRL, behavioral cloning does not model the underlying objective or reason about the consequences of actions. It suffers from distributional shift (compounding errors when the agent deviates from the expert's state distribution). IRL addresses this limitation by recovering the intent behind actions, enabling robust generalization to unseen states.