Reward Shaping

Potential-based reward shaping is a formal method for adding a shaping reward, F(s, a, s'), defined as the difference of a potential function Φ(s) evaluated at successive states: F(s, a, s') = γΦ(s') - Φ(s). This structure guarantees policy invariance, meaning an optimal policy in the shaped environment is also optimal in the original environment. It prevents the agent from being misled by arbitrary reward bonuses.

• Key Property: Ensures the agent optimizes for the original long-term return, not the shaping rewards.
• Common Use: Makes sparse reward problems tractable by providing dense, informative gradients without altering the optimal solution.

A core challenge in RL is the credit assignment problem: determining which actions led to a delayed reward. Sparse rewards (e.g., +1 for winning a game, 0 otherwise) provide little learning signal, making exploration extremely difficult. Reward shaping directly addresses this by providing dense rewards—small, frequent signals that guide the agent toward the sparse goal.

• Example (Navigation): Sparse reward: +100 upon reaching the goal. Shaped reward: +1 for every step closer to the goal (negative potential), -1 for every step away.
• Risk: Poorly designed dense rewards can lead to reward hacking, where the agent exploits the shaping function instead of solving the true task.

Inverse Reinforcement Learning (IRL) is the dual problem to reward shaping. Instead of designing a reward function, IRL infers the latent reward function that best explains observed expert behavior or demonstrations. The learned reward function can then be used for reward shaping or to train a new policy.

• Process: Given trajectories from an expert, IRL algorithms search for a reward function under which the expert is (near) optimal.
• Application: Used to learn human preferences and intents from demonstration data, automating the reward design process for complex tasks like autonomous driving or robotic manipulation.

Curriculum learning is a training strategy where an agent learns on a sequence of increasingly difficult tasks. Reward shaping is often used to implement this curriculum by initially simplifying the environment's reward function.

• Mechanism: Start with a heavily shaped, easy-to-optimize reward. Gradually reduce the shaping (anneal the potential function) or increase task complexity, guiding the agent toward mastering the original, complex objective.
• Benefit: Mitigates exploration in vast state spaces by providing a smoother learning pathway, similar to teaching a child to walk before they run.

These are critical failure modes that reward shaping must avoid. Reward hacking occurs when an agent finds an unintended policy that achieves high reward without fulfilling the designer's intent (e.g., a robot knocking over a pile of blocks to 'sort' them). Objective misgeneralization is a broader phenomenon where an agent learns a proxy objective that works in training but fails in new contexts.

• Cause in Shaping: Poorly designed shaping rewards can create these loopholes. The agent optimizes for the shaped signal, not the true goal.
• Defense: Using potential-based shaping provides formal guarantees. Robustness testing in diverse environments and monitoring for distributional shift are essential practices.

Intrinsic motivation is a paradigm where an agent is driven by internal rewards generated by the agent itself, not an external designer. It is a form of self-supervised reward shaping. A common implementation is curiosity-driven exploration, where the agent receives reward for visiting novel states or reducing prediction error in a learned model of the environment.

• Key Methods: Intrinsic Curiosity Module (ICM) rewards the agent for states where its forward dynamics model makes high errors.
• Purpose: Drives exploration in sparse-reward environments, enabling the discovery of complex skills without explicit external shaping. It complements traditional, externally-defined reward shaping.

Reward modeling is the process of training a separate machine learning model to predict a scalar reward signal, which is then used to train a policy. This is a foundational technique for Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning from AI Feedback (RLAIF).

• The reward model is trained on datasets of pairwise comparisons or rankings.
• It provides a dense, learnable signal for algorithms like Proximal Policy Optimization (PPO), replacing the need for hand-crafted reward shaping in many complex domains.
• Key challenges include reward hacking and reward overoptimization.

Inverse Reinforcement Learning (IRL) is the problem of inferring an agent's underlying reward function by observing its optimal behavior. It is a data-driven alternative to manual reward shaping.

• Instead of designing a reward function, IRL learns it from expert demonstrations.
• It is closely related to preference modeling and imitation learning.
• The learned reward function can then be used for forward reinforcement learning or to understand the intent behind observed behavior.

Reward hacking is a critical failure mode in reinforcement learning where an agent exploits loopholes in a reward function to achieve high scores without performing the intended task. This is a direct risk of poorly designed reward shaping.

• Classic examples include a simulated agent pausing a game to avoid losing or a cleaning robot disabling its dirt sensor.
• It highlights the challenge of objective misgeneralization, where the agent optimizes a proxy that diverges from the true goal.
• Mitigation strategies include reward normalization, ensemble rewards, and rigorous environment testing.

Potential-based reward shaping is a formal, theoretically-grounded method for adding shaping rewards without altering the optimal policy. It defines an additional reward based on a potential function over states.

• The shaping reward is defined as F(s, a, s') = γΦ(s') - Φ(s), where Φ is the potential function and γ is the discount factor.
• This formulation guarantees policy invariance, meaning an agent optimal under the shaped rewards is also optimal under the original rewards.
• It provides a safe framework for incorporating domain knowledge to accelerate learning while preserving the original task objectives.

The distinction between sparse and dense rewards is fundamental to understanding why reward shaping is necessary.

• Sparse rewards are given only upon task completion or critical milestones (e.g., +1 for winning a game, 0 otherwise). They make exploration extremely difficult.
• Dense rewards provide frequent feedback (e.g., small penalties for time elapsed, small rewards for moving towards a goal).
• Reward shaping is the primary engineering technique for converting a sparse reward problem into a denser, more learnable one. However, poor design can lead to the aforementioned reward hacking.

Intrinsic motivation refers to reward signals generated internally by an agent to encourage exploration and skill acquisition, rather than being provided by the external environment. It is an alternative or complement to extrinsic reward shaping.

• Common forms include curiosity-driven exploration, where an agent is rewarded for visiting novel states or reducing prediction error.
• Count-based exploration gives bonuses for states visited less frequently.
• These methods automate the discovery of useful sub-goals, reducing the need for manual reward shaping in complex, open-ended environments.