Reward Shaping

The central challenge in RL is credit assignment—determining which actions in a long sequence led to a final reward. A sparse reward (e.g., +1 only upon task completion) provides no guidance. Reward shaping introduces dense rewards that offer intermediate feedback, dramatically accelerating learning by helping the agent connect actions to outcomes.

• Example: In a maze, a sparse reward is only given upon exit. A shaped reward can provide a small positive signal for moving closer to the goal and a negative signal for moving away, creating a gradient the agent can follow.

Naive reward shaping can alter the optimal policy, causing the agent to learn a suboptimal behavior that maximizes shaped rewards instead of the true objective. Potential-based reward shaping is a formal solution. It defines the shaped reward as the change in a potential function Φ(s) defined over states:

R'(s, a, s') = R(s, a, s') + γΦ(s') - Φ(s)

Where γ is the discount factor. This formulation, derived from Lyapunov stability theory, guarantees that the optimal policy remains unchanged (policy invariance), making it a safe and theoretically sound shaping method.

Shaping rewards are often derived from domain knowledge or expert data.

• Inverse Reinforcement Learning (IRL): Infers a reward function from expert demonstrations, which can then be used as a shaped reward for a new agent.
• Heuristic Functions: Simple programmed rules provide guidance. For a robot stacking blocks, a heuristic could reward decreasing the distance between a block's current and target position.
• Learning from Play: Unstructured demonstration data (e.g., a human casually interacting with objects) can be used to learn a diverse reward function that encourages general, useful behaviors before a specific task is defined.

When environmental rewards are absent or extremely sparse, intrinsic motivation can serve as a powerful shaping signal. Curiosity is a common form, where the agent is rewarded for visiting novel states or for making predictions about environment dynamics that lead to high prediction error.

• Intrinsic Curiosity Module (ICM): A self-supervised auxiliary task where the agent learns a model of its environment and receives an intrinsic reward proportional to the model's prediction error in a learned feature space.
• Impact: This drives the agent to explore its environment thoroughly, often discovering the true extrinsic reward signal faster than random exploration.

Reward shaping is closely linked to curriculum learning. Instead of providing a single, complex reward from the start, the learning process is structured:

1. Start Simple: Provide shaped rewards for easy sub-tasks (e.g., reward a walking robot for not falling over).
2. Increase Complexity: Gradually reduce or remove the shaping rewards, forcing the agent to optimize for the final, more complex objective (e.g., walking in a specific direction).

This progression prevents the agent from becoming overly dependent on the shaped rewards and helps it learn robust policies for the true task.

Poorly designed shaping can lead to reward hacking, where the agent finds a loophole to maximize the shaped reward without solving the intended task.

• Example: An agent rewarded for 'holding an object' might learn to grasp it and never let go, failing a task that requires placing it elsewhere.
• Overfitting to Simulation: Shaping rewards tuned in a simulated environment may not translate to the real world (sim-to-real transfer), as the agent may exploit simulator physics not present in reality.
• Mitigation: Rigorous evaluation in environments with only the true, sparse reward is essential to validate that shaping has not corrupted the policy's objectives.

A classic example is robot navigation to a goal. A sparse reward provides +1 only upon reaching the goal and 0 otherwise, which is extremely difficult to learn from. Dense reward shaping provides incremental feedback:

• Potential-based shaping: Reward proportional to reduction in distance to goal.
• Step penalty: A small negative reward (e.g., -0.01) per timestep encourages efficiency.
• Collision penalty: A large negative reward (e.g., -1) for hitting obstacles. This transforms an almost impossible credit assignment problem into a learnable gradient of feedback.

Teaching a humanoid robot to walk involves shaping multiple competing objectives. The primary sparse reward might be forward velocity. Shaping adds auxiliary rewards for:

• Survival bonus: +0.1 for each timestep the torso remains upright (encourages stability).
• Joint limit penalty: Negative reward for exceeding safe actuator ranges.
• Energy efficiency penalty: Small negative reward proportional to torque squared to prevent wasteful, jittery movements.
• Foot contact reward: Positive reward for having at least one foot in contact with the ground, discouraging hopping. This composite reward function is standard in benchmarks like the MuJoCo Humanoid environment.

For a robotic arm tasked with picking up a block, the sparse success reward is insufficient. Shaping provides feedback on the grasping process:

• Distance to object: Reward inversely proportional to the gripper's distance to the block's center.
• Gripper alignment: Reward for orienting the gripper fingers parallel to the object's graspable surfaces.
• Contact reward: Small positive signal when fingers first make contact.
• Lift reward: Additional reward once the object is lifted above a threshold height. This breaks the monolithic 'pick up' task into a sequence of learnable sub-skills.

In a driving simulator, the goal is to stay in the lane. A shaped reward function might include:

• Lateral offset penalty: Continuous negative reward proportional to the car's distance from the lane center.
• Heading error penalty: Negative reward for the angular difference between the car's heading and the road's direction.
• Speed maintenance reward: Positive reward for maintaining a target speed, with penalties for exceeding it.
• Smoothness penalty: Negative reward for high jerk (change in acceleration) or steering rate to encourage passenger comfort. This creates a smooth optimization landscape compared to a binary 'crash/no-crash' reward.

Potential-Based Reward Shaping (PBRS) is a formal framework that guarantees the optimal policy remains unchanged. It defines the shaping reward F(s, a, s') as the difference in a potential function Φ(s): F(s, a, s') = γΦ(s') - Φ(s) Where γ is the discount factor. Example: For navigation, let Φ(s) be the negative distance to goal. The shaped reward becomes R + γ(-dist(s')) - (-dist(s)) = R + (dist(s) - γ*dist(s')). This rewards reducing discounted future distance. PBRS prevents the agent from being tricked by arbitrary shaping rewards (e.g., rewarding going in circles).

Reward shaping can dynamically change to create a learning curriculum. Initially, the reward function is heavily shaped to make early progress possible. As the agent improves, the shaping is gradually annealed, forcing the agent to rely more on the true, sparse objective. Example in Stacking Blocks:

1. Phase 1: Reward for moving near a block.
2. Phase 2: Reward for lifting a block.
3. Phase 3: Reward for placing a block near another block.
4. Phase 4: Reward only for a stable stack of N blocks. This reward annealing strategy is a practical method for overcoming exploration barriers in complex, long-horizon tasks.

The foundational mathematical framework for modeling sequential decision-making. An MDP is defined by a tuple (S, A, P, R, γ), where:

• S is a set of states.
• A is a set of actions.
• P(s'|s,a) is the transition probability function.
• R(s,a,s') is the reward function.
• γ is a discount factor. Reward shaping directly modifies the R function to provide denser, more informative feedback, making the underlying optimal policy easier for an agent to discover.

A formal, theoretically-grounded method for designing auxiliary rewards that guarantees policy invariance. The shaped reward is defined as: R'(s,a,s') = R(s,a,s') + γΦ(s') - Φ(s) where Φ(s) is a potential function defined over states.

• Key Property: This formulation ensures that the optimal policy under the shaped reward R' is also optimal under the original reward R.
• Application: Used to guide exploration by making promising states (high potential) more attractive without altering the final goal.

The inverse problem to reward shaping. Instead of designing a reward function, IRL infers the latent reward function that best explains observed expert behavior.

• Process: Given trajectories from an expert policy, an IRL algorithm learns a reward function under which the expert's behavior is (near) optimal.
• Relationship to Shaping: IRL can be used to automate reward design. The recovered reward function can then be used for training or serve as a starting point for further manual shaping.

A training strategy where an agent learns by progressing through a sequence of increasingly difficult tasks. It is closely related to reward shaping as both aim to ease the learning problem.

• Mechanism: Starts with a simplified version of the environment (e.g., easier dynamics, denser rewards) and gradually increases complexity.
• Contrast with Shaping: While reward shaping modifies the feedback signal within a single task, curriculum learning modifies the task definition itself. They are often used in conjunction.

This dichotomy defines the core problem reward shaping aims to solve.

• Sparse Reward: The agent receives a non-zero reward only upon task completion or critical milestones (e.g., +1 for winning a game, 0 otherwise). This leads to a hard exploration problem.
• Dense Reward: The agent receives frequent feedback correlated with progress (e.g., distance to goal, alignment with a desired pose). Reward shaping is the deliberate engineering process of transforming a sparse, non-informative reward signal into a dense, learnable one.

A class of methods where an agent generates its own internal reward signal to drive exploration in the absence of, or in addition to, external rewards.

• Common Forms: Curiosity (reward for visiting novel states), empowerment (reward for increasing control over the environment).
• Relation to Shaping: Intrinsic rewards are a form of automated, adaptive reward shaping. Unlike hand-designed shaping, intrinsic rewards are generated by the agent's own learning dynamics and often diminish as the environment becomes familiar.