Reward Function

A sparse reward provides a signal only upon task completion or failure (e.g., +1 for success, 0 otherwise). This is simple to design but makes exploration extremely difficult. A dense reward provides frequent, incremental feedback (e.g., negative distance to a goal). This guides learning but is prone to reward hacking, where the agent exploits loopholes to maximize reward without achieving the true objective. In robotics, shaping a dense reward from a sparse goal is a core engineering challenge.

The absolute magnitude and relative differences of reward values must be carefully calibrated. Rewards that are too large can cause instability during gradient-based training. The discount factor (γ) determines how much the agent values future rewards versus immediate ones. A function must be designed so that its discounted sum (the return) is finite and meaningful. For long-horizon robotics tasks, a discount factor close to 1 (e.g., 0.99) is typical to encourage planning.

While the RL agent interacts with a scalar reward, the underlying function should ideally be smooth and continuous with respect to state and action variables. Abrupt changes or discontinuities can create steep, hard-to-navigate optimization landscapes. In simulation, reward functions are often analytically defined and perfectly smooth. A key sim-to-real challenge is that the same function, when computed from noisy real-world sensors, may become non-smooth, disrupting policy performance.

Robotic tasks are rarely defined by a single goal. A reward function is often a weighted sum of multiple sub-rewards:

• Primary Objective: Reach goal location.
• Secondary Objectives: Minimize energy consumption, avoid obstacles, maintain stability.
• Safety Constraints: Penalize dangerous states (e.g., excessive joint torque). Tuning the weights for this linear scalarization is critical. Poor weighting can lead to the agent satisfying one objective at the severe expense of others.

The ideal reward is often a function of ground-truth state (e.g., true position). In reality, the agent only receives observations (e.g., noisy camera images). Therefore, the reward function must be computed from these same observations to be consistent. This often requires designing proxy rewards using estimated states. A mismatch between the training proxy (in simulation) and the real-world proxy is a major source of the reality gap.

A mis-specified reward function inadvertently rewards undesired behavior or fails to penalize critical failures. This is a fundamental alignment problem. Techniques to improve robustness include:

• Reward shaping: Adding potential-based shaping rewards to guide learning without altering the optimal policy.
• Inverse Reinforcement Learning (IRL): Inferring the reward function from expert demonstrations.
• Constrained RL: Using frameworks like Constrained Policy Optimization (CPO) to satisfy hard safety limits regardless of reward maximization.

A sparse reward provides a signal only upon task completion (e.g., +1 for success, 0 otherwise). This is simple but makes exploration extremely difficult. A dense reward provides incremental feedback at each timestep (e.g., negative distance to a goal). This guides learning but risks reward hacking, where the agent exploits the reward signal without achieving the true objective. In robotics, dense rewards are common in simulation but must be designed to correlate with real-world success.

Reward shaping adds an auxiliary reward to guide the agent toward a sparse goal. Naive shaping can alter the optimal policy. Potential-based reward shaping is a provably correct method where the shaping reward is defined as the difference of a potential function Φ(s) between states: r_shaped = r_env + γΦ(s') - Φ(s). This ensures the agent is still optimizing the original task. Example: Using the negative distance to a goal as a potential function for a navigation task.

Complex tasks require balancing multiple, often competing, objectives. A composite reward function is a weighted sum of sub-rewards: R(s,a) = w1 * r_task(s,a) + w2 * r_safety(s,a) + w3 * r_energy(s,a)

• r_task: Primary goal (e.g., grasp object).
• r_safety: Penalizes dangerous states (e.g., high joint torque).
• r_energy: Penalizes power consumption. Tuning the weights (w1, w2, w3) is a major engineering challenge. Constraint functions are an alternative, where safety objectives are enforced as hard limits rather than penalties.

Bipedal Locomotion: R = v_x - w_torque * ||τ||^2 - w_impact * ||F_impact|| - w_deviation * (θ - θ_stand)^2. Rewards forward velocity (v_x) while penalizing torque, foot impact forces, and deviation from a nominal standing pose. Robotic Manipulation (Pick-and-Place): R = -d_gripper_to_obj - d_obj_to_goal + 1[grasped] + 10[placed]. This dense reward minimizes distances, with sparse bonuses for successful grasp and placement. Autonomous Driving: R = v_progress - w_lateral * d_lane_center - w_speed * |v - v_desired| - 1000[collision]. Encourages progress while penalizing lane deviation, speed error, and collisions.

The reality gap means a reward function that works perfectly in simulation may fail on real hardware. Key design principles for transfer:

• Minimize Sim-Dependent Signals: Avoid rewards based on perfect simulation state (e.g., exact object pose) that isn't measurable in reality.
• Use Observable Quantities: Base rewards on sensor readings (e.g., joint encoders, camera pixels) available on the real robot.
• Add Robustness Penalties: Include terms that discourage policies overly sensitive to dynamics parameters, promoting policy robustness.
• Domain Randomization: Training with randomized physics parameters inherently shapes the reward landscape to favor transferable behaviors.

Inverse Reinforcement Learning is the process of inferring a reward function from demonstrations of expert behavior. It bypasses manual reward engineering. Given trajectories from an expert policy, IRL algorithms (like Maximum Entropy IRL) search for a reward function under which the expert is (near-)optimal. The output is a learned reward that can then be used to train a new policy via standard RL. This is closely related to Imitation Learning, but IRL recovers the intent (the reward), allowing for greater generalization than direct behavior cloning.

A value function estimates the expected cumulative future reward from a given state (V(s)) or state-action pair (Q(s,a)). It is the cornerstone for planning and policy evaluation, representing the long-term desirability of states, whereas the reward function provides the immediate, local feedback signal. The two are linked by the Bellman equation, which recursively defines the value of a state as the immediate reward plus the discounted value of the successor state.

Policy Gradient methods are a class of algorithms that directly optimize a parameterized policy function by ascending the gradient of expected reward. The reward function is critical here, as its signal directly shapes the gradient. High-variance rewards can destabilize training, leading to techniques like reward shaping or baseline subtraction. Algorithms like Proximal Policy Optimization (PPO) use the reward to compute advantages, which clip policy updates for stability.

This is the fundamental dilemma where an agent must balance trying new actions (exploration) to discover their effects with choosing known high-reward actions (exploitation). The design of the reward function directly influences this balance. Sparse rewards (e.g., +1 only on task success) necessitate extensive exploration, while dense, shaped rewards can guide exploitation more quickly but may lead to suboptimal local maxima if poorly designed.

Imitation Learning is a paradigm for training agents from expert demonstrations, bypassing the explicit design of a reward function. Common approaches include:

• Behavioral Cloning: Supervised learning on state-action pairs.
• Inverse Reinforcement Learning (IRL): Infers the underlying reward function that the expert is optimizing, then uses RL to learn a policy with that reward. This is particularly valuable in robotics where specifying a precise reward function for complex tasks like manipulation is exceptionally difficult.

Reward shaping is the technique of modifying the original reward function by adding supplemental rewards to guide the agent toward desired behaviors and accelerate learning. A key challenge is ensuring the shaping does not alter the optimal policy (preserving policy invariance). The Potential-Based Reward Shaping framework provides a formal method for creating shaping rewards that are guaranteed not to introduce new optimal behaviors, using the difference of a potential function across states.

Credit assignment is the problem of determining which actions in a long sequence are responsible for the eventual rewards received. A poorly timed reward signal makes this problem difficult. Temporal Difference (TD) Learning methods, like Q-learning, address this by propagating reward information backward through time via bootstrapping. The discount factor (gamma) in the reward formulation determines how much credit is assigned to recent versus distant actions.