Reward Ambiguity

Reward ambiguity arises because Inverse Reinforcement Learning (IRL) is an ill-posed inverse problem. Given a finite set of expert demonstrations, there are infinitely many reward functions under which those demonstrations appear optimal. This is because the mapping from reward functions to optimal policies is many-to-one; different reward structures can lead to the same observed behavior. For example, a robot moving to a goal could be maximizing proximity to the target, minimizing distance traveled, or avoiding specific regions—all resulting in similar-looking paths. This ambiguity must be resolved through additional assumptions or constraints to recover a unique, useful reward.

A primary source of ambiguity is the scale invariance of reward functions in Markov Decision Processes (MDPs). An optimal policy is unchanged if the reward function R(s, a) is multiplied by any positive constant or if a state-dependent constant is added. Formally, for any function φ(s), the transformed reward R'(s, a, s') = R(s, a, s') + γφ(s') - φ(s) (where γ is the discount factor) induces the same optimal policy. This is known as potential-based reward shaping. Consequently, IRL algorithms can only recover a reward function up to this equivalence class, making the true "intent" behind the demonstrations fundamentally underdetermined.

The ambiguity is compounded by the maximization operator in the definition of an optimal policy (π* = argmax E[Σ γ^t R_t]). The agent only needs to distinguish between actions that are optimal versus suboptimal; the precise numerical difference in reward values between a chosen optimal action and a rejected suboptimal one is not uniquely specified. As long as the reward function maintains the same ordering of action-values (Q-values) in each state, the optimal policy remains identical. This means IRL algorithms often recover a reward that explains the choice but not the exact preference strength the expert might have had.

Ambiguity increases if the learner's model of the expert's state representation or observation space is incorrect. The expert may be optimizing a reward based on features or state variables that the learner does not have access to or has misidentified. For instance, a human might open a door gently to avoid noise, but a robot observing only kinematic data misses the acoustic feature. The learner might then attribute the gentle motion to a different cause, inferring an incorrect reward for smoothness. This form of ambiguity links reward recovery to the challenge of state abstraction and feature selection.

To resolve ambiguity, IRL algorithms impose regularization or Bayesian priors to select a single, plausible reward function from the feasible set. Common approaches include:

• Maximum Entropy IRL: Selects the reward function that maximizes the entropy of the distribution over trajectories, yielding the least committed explanation.
• Feature Expectation Matching: Assumes the reward is a linear combination of known features and matches the expected feature counts of the learner to the expert.
• Sparsity Priors: Prefer reward functions with few non-zero weights (e.g., via L1 regularization), based on the principle that experts optimize for simple objectives.
• Gaussian Processes / Deep Networks: Use flexible function approximators with appropriate inductive biases to generalize from limited demonstrations.

The specific reward function chosen from the ambiguous set critically impacts policy generalization to new states or environments. A reward that merely matches demonstrations in the training distribution may fail if the agent encounters novel situations. For example, an ambiguously recovered reward for a self-driving car might not properly penalize a rare obstacle. Therefore, resolving ambiguity isn't just an academic exercise; it directly affects the robustness and safe deployment of the imitated policy. Techniques like robust IRL seek rewards that lead to good performance under a worst-case transition dynamics or a distribution of environments.

Inverse Reinforcement Learning (IRL) is the foundational technique where the problem of reward ambiguity originates. Its goal is to infer an unknown reward function from observed optimal behavior (demonstrations). The core assumption is that the expert's actions are optimal with respect to some latent reward function the algorithm must recover.

• Core Problem: Given trajectories τ ~ π_expert, find R such that π_expert is optimal for R.
• Ambiguity Source: Many reward functions (e.g., R(s,a) = 1, R(s,a) = 0) can make the same policy appear optimal, leading to the fundamental ill-posedness of IRL.

Maximum Entropy Inverse Reinforcement Learning is a seminal probabilistic framework designed explicitly to resolve reward ambiguity. Instead of finding a single reward function, it finds a distribution over trajectories where those with higher reward are exponentially more likely, but all trajectories have non-zero probability.

• Key Principle: It chooses the reward function that maximizes the entropy of the distribution over expert trajectories, making the fewest additional assumptions.
• Resolves Ambiguity: By not committing to a single optimal path, it finds the reward function that best explains the distribution of expert behavior, not just a single optimal policy. This often results in smoother, more generalizable reward functions.

Inverse Optimal Control is the classical control theory counterpart to IRL, focusing on deterministic, often continuous-time systems. Given optimal trajectories, IOC aims to recover the cost function (the negative of a reward function) the agent was minimizing.

• Domain: Historically applied in robotics, biomechanics, and econometrics with known system dynamics.
• Relation to Ambiguity: Like IRL, IOC faces the same fundamental ambiguity—many cost functions can produce the same optimal trajectory. Resolution often comes from imposing structural priors (e.g., quadratic cost) or using the maximum entropy principle.

Preference-Based Reward Learning is a technique that sidesteps reward ambiguity by learning from comparative human judgments instead of optimal demonstrations. A human is queried for preferences between trajectory segments ("Which outcome looks better?").

• Data Source: Pairwise (or ranked) preferences over trajectories, not assumed-optimal demonstrations.
• Mitigates Ambiguity: By learning from relative comparisons, the algorithm recovers a reward function that explains the ordering of outcomes, which is often less ambiguous than trying to explain absolute optimality. Frameworks like Bradley-Terry models are commonly used to model these preferences.

Generative Adversarial Imitation Learning is an adversarial method that bypasses explicit reward function inference. Instead of solving the ambiguous IRL step, it directly learns a policy whose state-action distribution matches the expert's.

• Mechanism: A policy (generator) is trained to fool a discriminator that tries to distinguish between expert and learner trajectories.
• Relation to Ambiguity: GAIL implicitly performs distribution matching without ever recovering a unique reward function, thus avoiding the ill-posed IRL problem altogether. It demonstrates that successful imitation does not require disambiguating the true reward.

The presence of suboptimal demonstrations exacerbates the challenge of reward ambiguity. If demonstrations contain mistakes or are not perfectly optimal, the core IRL assumption is violated, making the true reward function even harder to identify.

• Impact on Ambiguity: With imperfect data, an even larger set of reward functions becomes plausible (some that explain the optimal parts, others that explain the mistakes).
• Algorithmic Response: Modern IRL and imitation learning methods incorporate robustness to suboptimality, for example by modeling the expert as noisily rational (e.g., using a Boltzmann distribution) or by jointly learning a reward function and filtering out poor demonstrations.