Reward Shaping

A mathematically grounded technique that adds a shaping reward defined as the difference in a potential function Φ(s) between successive states: F(s, a, s') = γΦ(s') - Φ(s). This method guarantees policy invariance, meaning an optimal policy under the shaped rewards is also optimal under the original sparse rewards. It prevents the agent from exploiting the shaping rewards through cyclic behavior.

• Key Property: Preserves the optimal policy, ensuring the shaping guides learning without changing the final goal.
• Common Use: In navigation tasks, Φ(s) could be the negative distance to the goal.

The practice of designing a reward function that provides frequent, granular feedback for every step towards a goal, rather than a single reward upon completion. This transforms a sparse reward problem into a dense reward problem, dramatically improving learning speed.

• Examples: Giving a small positive reward for decreasing the distance to a target object, or a negative reward for each timestep elapsed.
• Challenge: Poorly designed dense rewards can lead to reward hacking, where the agent finds unintended ways to maximize reward without solving the intended task (e.g., circling near a target instead of touching it).

A training paradigm where the task difficulty is gradually increased. Initial reward functions are shaped to be simple and forgiving, guiding the agent to learn basic skills. As the agent masters these, the reward function and environment complexity are progressively scaled towards the final, challenging objective.

• Process: 1) Learn to reach any point in an empty room. 2) Learn to reach a specific point. 3) Learn to reach a point while avoiding static obstacles. 4) Learn with dynamic obstacles.
• Benefit: Avoids the agent being overwhelmed by the complexity and sparsity of the final task from the outset.

A technique for reward shaping by demonstration. Instead of manually engineering a reward function, IRL infers the underlying reward function from observations of expert behavior (e.g., human teleoperation of a robot). The learned reward function captures the implicit preferences and constraints of the expert.

• Application: Used when the true goal is complex or hard to specify (e.g., "drive politely") but can be demonstrated.
• Outcome: Produces a shaped reward function that can then be used to train a policy via standard RL, often generalizing better than direct imitation learning.

Augments the extrinsic task reward with an intrinsic reward generated by the agent itself to encourage exploration. This is a form of automatic reward shaping critical for environments with no or very sparse extrinsic rewards.

• Prediction Error: Rewards the agent for visiting states where its internal model makes poor predictions (i.e., novel or surprising states).
• Count-Based: Assigns higher reward to states visited less frequently.
• Impact: Drives the agent to comprehensively explore its state space, often discovering the extrinsic reward by accident, which then guides further learning.

Shapes learning by defining multiple, simpler reward signals for auxiliary tasks that are correlated with the main objective. The agent learns from a weighted sum of these rewards or by switching between tasks.

• Example: A robot arm's main task is to assemble a part. Auxiliary rewards can be given for: 1) Gripping the part securely (force sensor reading). 2) Moving the part closer to the assembly point. 3) Aligning the part correctly (orientation).
• Benefit: Provides learning gradients even when the main task reward is distant, and can lead to more robust policies that satisfy multiple constraints.

In tasks like pick-and-place or door opening, the final reward (successful placement) is extremely sparse. Shaping provides dense, incremental feedback.

• Distance-to-Goal Reward: A negative reward proportional to the Euclidean distance between the gripper and the target object, encouraging the agent to move closer.
• Orientation Penalty/Bonus: A reward component based on the angular difference between the gripper's orientation and the optimal grasp angle.
• Contact Detection: A small positive reward for making physical contact with the target object, marking progress toward the final grasp.

Without shaping, the agent receives zero learning signal until the unlikely event of a perfect grasp occurs by random exploration.

Teaching a bipedal or quadrupedal robot to walk involves shaping a reward function that promotes stable, efficient, and directed movement.

• Forward Velocity Tracking: A primary shaped reward encouraging the robot's center-of-mass velocity to match a target speed.
• Survival Bonus: A small positive reward for each timestep the robot remains upright, preventing early termination. This directly counters sparse failure penalties.
• Energy Efficiency Penalty: A negative reward proportional to the sum of squared joint torques, discouraging wasteful, jittery movements.
• Smoothness Penalty: A penalty on the jerk (rate of change of acceleration) of joint movements to produce natural, stable gaits.

This combination guides the policy away from falling and toward efficient, directed locomotion.

In navigation tasks, the only natural reward is often a large bonus upon reaching the goal. Shaping creates a potential field that guides the agent.

• Potential-Based Reward Shaping (PBRS): A canonical method where the shaping reward is defined as F(s, a, s') = γ * Φ(s') - Φ(s). Here, Φ(s) is a potential function, often defined as the negative distance from state s to the goal. This guarantees policy invariance—the optimal policy is unchanged.
• Example: If Φ(s) = -distance_to_goal(s), the agent gets a positive reward for any action that reduces the distance to the goal.
• Sparse Danger Penalties: Large negative rewards for colliding with obstacles provide clear, sparse negative feedback, while PBRS provides continuous positive guidance.

Reward shaping is heavily used in physics-based simulation to train policies that will transfer to real hardware. The shaped reward must produce robust behaviors that are insensitive to the reality gap.

• Domain Randomization-Aware Shaping: Rewards are designed to be based on features that are consistent between sim and real (e.g., joint angles, contact states) rather than raw physics values that may differ.
• Penalizing Sim-Artifact Exploits: Agents often find "cheats" in simulation physics. Shaping includes penalties for behaviors like high-frequency jitter or exploiting unrealistic friction, which would fail on real robots.
• Training for Robustness: Instead of a sharp reward for being exactly at a target, a shaped reward with a wider tolerance basin is used (e.g., a reward that plateaus near the goal), producing policies less sensitive to small positional errors in the real world.

In Hierarchical Reinforcement Learning, a high-level manager sets subgoals for a low-level worker. Reward shaping is intrinsic to this architecture.

• Manager Reward: The manager receives a sparse extrinsic reward upon ultimate task completion. Its subgoal selection is shaped by the worker's success.
• Worker Reward (Shaped): The low-level policy's reward is entirely shaped. It receives a dense reward based on how closely it achieves the manager's current subgoal (e.g., reduce distance to subgoal position).
• Example - Multi-Room Navigation: The manager's goal is "reach the final room." It first sets the subgoal "reach the doorway." The worker gets a shaped reward for moving toward the doorway. This decomposes a long-horizon sparse task into a series of short-horizon dense tasks.

Reward functions can be dynamically altered throughout training as part of a curriculum, starting simple and increasing in complexity.

• Phase 1 - Basic Stability: For a walking robot, the initial reward function might heavily weight a simple "stay upright" bonus, with minimal shaping for movement.
• Phase 2 - Introduce Direction: Once stable, the reward is reshaped to include a strong component for moving forward.
• Phase 3 - Refine Efficiency: Finally, energy penalties and smoothness terms are added to the reward to polish the gait.
• Automated Curriculum: The transition between phases can be triggered automatically by agent performance, a technique known as goal-conditioned RL or self-paced learning. The shaped reward evolves with the agent's capability.

The sparse reward problem occurs when an agent receives a meaningful reward signal only upon rare, successful completion of a complex task (e.g., a robot solving a Rubik's cube). This makes learning extremely inefficient, as most actions provide no feedback. Reward shaping is a primary solution to this problem by providing dense, intermediate rewards that guide the agent toward the goal.

Inverse Reinforcement Learning (IRL) is the process of inferring an unknown reward function from observations of expert behavior. While reward shaping manually engineers a reward, IRL learns it. This is crucial for robotics when:

• The true objective is complex to specify (e.g., "drive politely").
• Demonstrations are available but a reward function is not. IRL provides a data-driven alternative to manual reward shaping.

Potential-based reward shaping is a formal, theoretically grounded method for adding shaping rewards. It defines an auxiliary reward as the difference in a potential function Φ(s) evaluated at successive states: F(s, a, s') = γΦ(s') - Φ(s). This method guarantees that the optimal policy is preserved—the agent is guided but not misled. It provides a safe framework for implementing heuristic guidance without creating reward hacking scenarios.

Intrinsic motivation involves generating internal reward signals to drive exploration, independent of the external task reward. It tackles exploration challenges in sparse reward environments. Common forms include:

• Curiosity-driven: Reward for reducing prediction error in a learned dynamics model.
• Count-based: Reward for visiting novel or infrequent states. In robotics, intrinsic motivation is often used in conjunction with shaped extrinsic rewards to ensure both exploration and goal-directed behavior.

Reward hacking is a failure mode where an RL agent exploits flaws or unintended loopholes in a reward function to achieve high reward without performing the intended task. Poorly shaped rewards are a common cause. Classic examples include a simulated robot receiving reward for "standing up" learning to repeatedly fall and stand, or a cleaning robot learning to hide dirt instead of removing it. This underscores the need for robust, tested reward functions.

Curriculum learning is a training strategy where an agent learns by progressing through a sequence of increasingly difficult tasks. Instead of shaping the reward for a single complex task, the environment or task definition itself is shaped. For robotics, this might involve:

• Starting with object grasping in a fixed position, then adding clutter.
• Training a walking robot first on flat ground, then on slopes. It is a complementary approach to reward shaping for managing learning complexity.