Reward Function

This characteristic defines the frequency of feedback. A dense reward provides frequent, incremental feedback (e.g., +0.1 for moving towards a goal), which eases credit assignment but risks reward hacking. A sparse reward provides feedback only upon task completion or critical milestones (e.g., +1 for winning, 0 otherwise), which is more natural but makes exploration exponentially harder. Most practical systems use reward shaping to transform a sparse objective into a denser, learnable signal without altering the optimal policy.

The reward function must enable the agent to correctly attribute long-term outcomes to earlier actions. This is the credit assignment problem. A well-designed reward provides a clear causal link. Poorly timed rewards can lead to delayed consequences, where a catastrophic failure is only penalized many steps after the causative action. Techniques like discounted return and advantage estimation in algorithms are direct responses to this challenge, but the underlying reward signal must be temporally informative.

The absolute scale of rewards influences numerical stability during training. Extremely large values can cause exploding gradients, while very small values can lead to vanishing updates. Consistent reward scaling or normalization (e.g., dividing by a running estimate of standard deviation) is a standard engineering practice. Furthermore, the relative magnitude of positive vs. negative rewards defines the agent's risk tolerance. A large penalty for failure may encourage overly cautious behavior.

A critical failure mode where an agent discovers a policy loophole to maximize cumulative reward without achieving the intended goal. This occurs when the reward function is an imperfect proxy for the true objective. Classic examples include a vacuum-cleaning robot earning reward for dumping dirt instead of collecting it, or a boat-racing agent circling to repeatedly hit reward tokens. Mitigation involves reward validation through human oversight, multi-component reward functions with penalty terms, and adversarial reward validation.

Real-world tasks often involve balancing competing goals (speed vs. safety, accuracy vs. cost). A reward function can combine multiple objectives into a scalarized reward (e.g., R_total = w1R_speed + w2R_safety). For hard constraints (e.g., "never crash"), the reward can be structured with a barrier function that applies a large negative penalty upon violation. More advanced frameworks like Constrained MDPs (CMDPs) treat constraints separately from the reward, optimizing policy under explicit limits.

A reward function should incentivize behavior that is robust to variations in the environment. An agent trained with a reward tightly coupled to specific visual textures or object positions may fail if these change. Techniques like domain randomization in simulation often extend to the reward function itself, where reward calculation parameters are slightly randomized during training. This encourages the agent to learn the underlying principle of the task rather than overfitting to a precise reward signal.

A sparse reward provides feedback only upon task completion or critical milestones, offering no guidance during intermediate steps. This is common in goal-based environments.

• Example: Maze Navigation: Reward = +1 for reaching the goal, 0 for all other states.
• Challenge: Creates a hard exploration problem, as the agent receives no signal until it accidentally succeeds.
• Use Case: Intrinsic motivation or curiosity-driven exploration is often required to overcome this.

A dense reward provides frequent, granular feedback, often designed to guide the agent incrementally toward the goal. Reward shaping manually adds intermediate rewards to ease learning.

• Example: Robotic Manipulation: Reward = -(distance from gripper to object) + -(distance from object to goal).
• Benefit: Provides a smoother gradient for optimization, often accelerating learning.
• Risk: Poor shaping can lead to reward hacking, where the agent exploits loopholes to maximize reward without solving the intended task.

The reward function primarily defines costs or penalties for undesirable behaviors, framing the problem as cost minimization. The agent's goal is to accumulate the least negative reward.

• Example: Autonomous Driving: Reward = -0.1 for every timestep (encourage efficiency), -10 for colliding, -5 for lane departure.
• Use Case: Common in safe reinforcement learning and real-world systems where constraint violation must be strongly discouraged.
• Design Principle: Penalties must be scaled appropriately to balance different types of failures.

A composite reward is a weighted sum of multiple sub-rewards, each quantifying progress toward a different sub-goal. This formalizes trade-offs in multi-objective optimization.

• Example: Video Game Agent: Reward = +1 for defeating enemy, +0.01 for collecting coin, -0.1 for losing health.
• Design Challenge: Tuning the weights (e.g., w1*R1 + w2*R2) is critical and often requires domain expertise or automated search.
• Advanced Method: Constrained RL (e.g., Constrained Policy Optimization) explicitly handles objectives as constraints rather than blending them into a single scalar.

Intrinsic rewards are generated by the agent itself to encourage exploration or skill learning, independent of the external task reward. They are added to the extrinsic reward signal.

• Curiosity: Reward for visiting novel or unpredictable states (e.g., based on prediction error of a learned dynamics model).
• Example: R_intrinsic = η * ||φ(s_{t+1}) - f(φ(s_t), a_t)||^2, where f is a learned forward model.
• Purpose: Drives exploration in sparse-reward environments and prevents the agent from getting stuck in uninformative states.

In Inverse Reinforcement Learning (IRL), the reward function is inferred from expert demonstrations, under the principle that the observed behavior is approximately optimal. The learned reward is then used to train a new policy.

• Process: Given expert trajectories τ_E, an IRL algorithm finds a reward function R such that the expert's policy appears optimal under R.
• Use Case: Applicable when designing an explicit reward function by hand is difficult, but expert behavior can be demonstrated.
• Outcome: The resulting reward function often captures the intent behind the expert's actions, not just superficial features.

The foundational mathematical framework for modeling sequential decision-making. An MDP is formally defined by a tuple (S, A, P, R, γ):

• S: A set of states.
• A: A set of actions.
• P: The state transition probability function, P(s'|s, a).
• R: The reward function, R(s, a, s'), which provides the scalar feedback signal.
• γ: A discount factor, γ ∈ [0, 1], determining the present value of future rewards. The reward function R is a core component of this tuple, defining the agent's goal within the MDP's structure.

A value function estimates the long-term expected return (cumulative discounted reward) from a given state or state-action pair. It is the primary quantity that RL algorithms learn to optimize.

• State-Value Function V(s): The expected return starting from state s and following policy π.
• Action-Value Function Q(s, a): The expected return starting from state s, taking action a, and thereafter following policy π. The reward function provides the immediate signal (R) that is summed and discounted to compute these expected returns. Learning a value function is essentially learning to predict the future output of the reward function under a given policy.

A policy, denoted π(a|s), is the agent's strategy—a mapping from states to probabilities of selecting each possible action. The policy is what the agent learns to optimize.

• The reward function provides the criteria for optimization (what is good/bad).
• The policy is the solution that maximizes the cumulative reward. Algorithms directly adjust the policy based on rewards received, making the reward function the ultimate guide for policy search. A poorly shaped reward function can lead to the policy learning unintended or harmful behaviors.

Credit assignment is the challenge of determining which actions in a long sequence are responsible for the final rewards received. The reward function directly influences this problem.

• Sparse rewards (e.g., +1 only upon task completion) make credit assignment extremely difficult, as most actions receive no feedback.
• Dense rewards (frequent, small rewards) provide more guidance but require careful design to avoid reward hacking, where the agent exploits loopholes to maximize reward without solving the intended task. Techniques like reward shaping are explicitly designed to modify the reward function to ease the credit assignment problem.

Reward shaping is the practice of adding auxiliary rewards to the primary reward function to guide the agent toward desired behaviors and accelerate learning. A shaped reward R' = R + F, where R is the primary reward and F is a shaping potential.

• Potential-based shaping guarantees that the optimal policy remains unchanged, using a function of the state F(s).
• It is a critical engineering technique to overcome sparse rewards and difficult exploration, but requires domain expertise to design without introducing unintended incentives.

Inverse Reinforcement Learning (IRL) is the dual problem: inferring the underlying reward function from observed optimal behavior (expert demonstrations).

• Assumption: The observed behavior is optimal with respect to some unknown reward function.
• Goal: Recover the reward function R that best explains why the demonstrated trajectories are optimal.
• Application: Used in imitation learning to recover a reward function that can then be used to train a robust policy, often more generalizable than directly cloning the expert's actions.