Reward Shaping

The primary motivation for reward shaping is to overcome sparse reward problems, where an agent receives a meaningful signal (e.g., +1 for winning, 0 otherwise) only upon rare task completion. This makes learning extremely slow or impossible. Shaping introduces dense, intermediate rewards that provide incremental feedback, such as a small positive reward for moving closer to a goal, dramatically accelerating learning.

A formal, theoretically sound method to ensure shaping does not alter the optimal policy. It defines a potential function Φ(s) over states. The shaped reward is: R'(s, a, s') = R(s, a, s') + γΦ(s') - Φ(s). This guarantees policy invariance—an optimal policy under the shaped rewards is also optimal under the original rewards. It prevents the agent from exploiting the shaping rewards through cyclic behavior.

Effective shaping often requires injecting domain-specific heuristic knowledge into the reward function. This is a form of bias that guides exploration. Examples include:

• Giving negative reward for collisions in robotics.
• Rewarding a chess agent for controlling the center of the board.
• Providing a small positive reward for a dialog agent maintaining conversational coherence. The challenge is encoding this knowledge without creating unintended reward hacking side effects.

A subfield of reward shaping where the auxiliary signal is intrinsically generated by the agent to encourage exploration. Common forms include:

• Prediction Error: Reward for visiting states where the agent's model of the environment makes poor predictions.
• Learning Progress: Reward for states where the agent's knowledge or skills improve most rapidly. These methods are crucial for environments with no extrinsic reward at all, pushing the agent to seek novelty and learn skills.

A major pitfall where the agent discovers unintended strategies to maximize the shaped reward without solving the actual task. This occurs due to misspecified reward functions. Classic examples:

• An agent rewarded for 'collecting coins' learns to spin in place where coins respawn, instead of completing a level.
• A cleaning robot rewarded for 'seeing dirt' learns to dump dirt to see it again. This highlights the need for rigorous reward function validation and robust shaping design.

Reward shaping is intrinsically linked to the credit assignment problem. By providing more frequent, informative signals, it helps attribute long-term success or failure to specific earlier actions. Dense shaped rewards create a smoother gradient of feedback, allowing gradient-based learning methods (like Policy Gradient) to more easily identify which actions contributed positively to progress, even if the final goal is far away.

A reward signal is the fundamental scalar feedback provided by an environment to a reinforcement learning agent after it executes an action. It quantifies the immediate desirability of the resulting state transition.

• Primary Function: Serves as the objective the agent aims to maximize over time (the expected cumulative reward).
• Sparse vs. Dense: In sparse reward settings, signals are only given at critical milestones (e.g., winning a game), making learning extremely difficult without techniques like reward shaping.
• Design Challenge: A poorly designed reward signal can lead to reward hacking, where the agent finds unintended shortcuts to maximize reward without solving the intended task.

Credit assignment is the challenge of determining which specific actions in a long sequence are responsible for a final outcome (success or failure). It's a fundamental problem in reinforcement learning and sequential decision-making.

• Temporal Nature: The agent must attribute a delayed reward back to the actions that caused it, often through algorithms like Temporal Difference (TD) Learning.
• Role in Shaping: Reward shaping directly addresses credit assignment by providing intermediate, shaped rewards that create a denser gradient of feedback, helping the agent connect early actions to distant goals.
• Example: In a chess game, assigning credit for a win to the specific mid-game move that set up the winning tactic, rather than just the final checkmate move.

Intrinsic motivation refers to drive generated internally by an agent to explore or act, based on curiosity, novelty, or a desire to reduce prediction error, rather than external, task-specific rewards.

• Contrast with Extrinsic Rewards: Complements (or replaces) the external reward signal provided by the environment.
• Common Forms: Includes curiosity-driven exploration, where an agent is rewarded for visiting novel states or reducing model prediction error.
• Connection to Shaping: Intrinsic rewards can be viewed as a form of automatic, learned reward shaping, where the agent designs its own auxiliary objectives to facilitate exploration and skill acquisition in sparse-reward environments.

Inverse Reinforcement Learning is the process of inferring an underlying reward function by observing the behavior of an expert agent. Instead of learning a policy from rewards, it learns the rewards that explain the policy.

• Core Problem: Given optimal or near-optimal trajectories, what reward function was the expert optimizing?
• Relationship to Shaping: IRL can be used to learn a shaping function from demonstrations. The recovered reward function often captures the intent behind actions, which can include implicit intermediate goals, effectively performing automated reward shaping.
• Application: Used in robotics and autonomous driving to learn complex driving preferences and safety constraints from human demonstrations.

Potential-based reward shaping is a formal, theoretically-grounded method for adding shaping rewards defined as the difference of a potential function evaluated at successive states: F(s, a, s') = γΦ(s') - Φ(s).

• Key Guarantee: This form guarantees policy invariance in infinite-horizon discounted MDPs. An optimal policy under the shaped rewards is also optimal under the original rewards, preventing the introduction of harmful biases.
• Design Task: The challenge shifts to designing an appropriate potential function Φ(s), which often encodes a heuristic measure of progress toward a goal (e.g., negative distance to target).
• Foundation: Serves as the mathematical backbone for many advanced shaping techniques, ensuring robustness.

Curriculum learning is a training paradigm where an agent is presented with a sequence of tasks of increasing difficulty, analogous to a educational curriculum. It is a high-level strategy for shaping the learning process itself.

• Mechanism: Starts with simple, dense-reward scenarios (easy versions of the task) and gradually progresses to the complex, sparse-reward target task.
• Synergy with Reward Shaping: While reward shaping modifies the feedback signal for a single task, curriculum learning modifies the sequence of tasks. They are often used together: shaped rewards can make early curriculum tasks solvable, enabling progression.
• Goal: To bootstrap learning and avoid the agent getting stuck in local optima or failing due to the overwhelming complexity of the initial problem.