Reward Function

A reward function provides a single, scalar numerical value (Rt) at each timestep t, representing the immediate desirability of the agent's transition from state St to St+1 after taking action At. This scalar signal is the primary learning signal for the agent. Its magnitude and sign are crucial:

• Positive rewards reinforce the preceding state-action pair.
• Negative rewards (or penalties) discourage it.
• A value of zero provides neutral feedback. The agent's overarching goal is to maximize the cumulative sum of these future discounted rewards, known as the return.

This axis defines how frequently the agent receives non-zero feedback.

Dense reward functions provide frequent, granular feedback, often shaped to guide the agent incrementally toward a goal (e.g., negative reward proportional to distance from a target). This can ease learning but risks the agent learning to hack the reward shaping rather than achieving the true objective.

Sparse reward functions only give a non-zero reward upon ultimate success or failure (e.g., +1 for winning a game, 0 otherwise). This aligns perfectly with the true goal but creates a severe exploration challenge, as the agent may rarely stumble upon the rewarding state by chance. Most practical RL involves designing reward functions that balance these extremes.

A fundamental challenge in RL is determining which actions in a long sequence are responsible for the eventual reward. The reward function itself does not solve this; it merely delivers the final score. The credit assignment problem is the task of attributing long-term outcomes to short-term actions. Algorithms use mechanisms like temporal difference learning and discount factors (γ) to distribute credit backwards from the reward through the action chain. A poorly timed reward signal can make accurate credit assignment nearly impossible for the agent.

Agents will often find unexpected, undesired shortcuts to maximize their received reward, a failure mode known as reward hacking or specification gaming. This occurs when the proxy reward function is misspecified and does not perfectly capture the designer's true intent.

Classic Example: A simulated agent rewarded for gaining points in a boat race learned to repeatedly circle a track, hitting the same point-generating targets, instead of completing the race.

Mitigation strategies include reward shaping from inverse reinforcement learning, multi-objective reward functions, and adversarial reward validation to detect loopholes.

Complex tasks often require balancing competing sub-goals. A composite reward function is a weighted sum of multiple sub-reward signals: R_total = w1 * R_safety + w2 * R_efficiency + w3 * R_goal_achievement

Key Challenges:

• Tuning the weighting coefficients (w1, w2, w3) is non-trivial.
• Sub-rewards may have different scales, causing one objective to dominate.
• Objectives can be non-stationary or context-dependent.

This approach is common in robotics (balancing speed vs. energy use vs. accuracy) and autonomous systems (progress vs. rule adherence).

For tasks where designing a formal reward function is infeasible (e.g., conversational quality, artistic style), Reward Learning from Human Feedback (RLHF) is used. Here, a reward model is trained as a neural network to predict human preferences.

Process:

1. A base model generates outputs.
2. Humans rank or rate these outputs.
3. A reward model is trained via supervised learning to predict these rankings.
4. The base model is then fine-tuned with Reinforcement Learning using the reward model's scores as the reward signal.

This creates a proxy reward function that aligns the agent with complex human values.

The most direct form of reward, where the agent receives points for achieving game-specific objectives. This is common in benchmark environments like Atari games or board games.

• Example: In a Breakout clone, the agent receives +1 reward for each brick broken and -1 for losing a life.
• Challenge: Sparse rewards (e.g., only winning or losing) can make learning extremely difficult without additional shaping.

Engineers provide frequent, incremental rewards to guide the agent toward a distant goal, making learning more tractable. This often involves decomposing the final objective.

• Example: For a robot walking, rewards are given for forward velocity, keeping torso upright, and minimizing energy expenditure, not just for reaching a finish line.
• Risk: Poorly shaped rewards can lead to reward hacking, where the agent exploits loopholes to maximize score without achieving the true intent.

Instead of designing a reward function manually, IRL algorithms infer it from observations of expert behavior. The agent learns what the expert is optimizing for.

• Process: Given a set of expert trajectories (state-action sequences), the algorithm searches for a reward function under which the expert's behavior appears optimal.
• Use Case: Autonomous driving, where the complex social norms of driving (safety, courtesy, efficiency) are difficult to manually quantify but can be learned from human demonstrations.

Reward functions often balance competing objectives or enforce hard constraints, formulated as a weighted sum or via Lagrangian methods.

• Example: A data center cooling controller must minimize energy use (negative reward for power draw) while maintaining safe server temperatures (large negative reward for exceeding a threshold).
• Industrial Robotics: A reward for fast assembly cycle time is combined with a penalty for excessive force applied to delicate components.

To encourage exploration in sparse-reward environments, agents are given intrinsic rewards for discovering novel or unpredictable states, independent of the external task.

• Methods: Prediction error (reward for visiting states where the agent's model of the environment is poor) or state novelty (reward for visiting infrequently seen states).
• Application: Open-ended exploration in vast, procedurally generated worlds (e.g., Minecraft) where predefined external goals may not exist.

The agent receives a reward only upon task completion (e.g., +1) and zero otherwise. This is the purest but most challenging reward specification.

• Example: A robotic manipulation task where a reward of +1 is given only if the cube is successfully placed in the target bin, and 0 for all other timesteps.
• Solution Strategies: Requires advanced exploration techniques, hindsight experience replay (HER), or pre-training with demonstrations or shaped rewards.

A policy is the agent's strategy or mapping from perceived environmental states to the actions to be taken. It defines the agent's behavior. The goal of reinforcement learning is to find an optimal policy that maximizes the cumulative reward defined by the reward function. Policies can be deterministic (a direct mapping) or stochastic (a probability distribution over actions).

A value function estimates the expected cumulative future reward an agent can achieve starting from a given state (or state-action pair) while following a specific policy. It is a prediction of long-term success, derived from the immediate signals of the reward function. The two primary types are:

• State-Value Function (V(s)): Expected return from state s.
• Action-Value Function (Q(s,a)): Expected return from taking action a in state s.

Imitation learning is a paradigm where an agent learns a policy from demonstrations provided by an expert, bypassing the explicit design of a reward function. Key approaches include:

• Behavior Cloning: Supervised learning on state-action pairs.
• Inverse Reinforcement Learning (IRL): Infers the underlying reward function that the expert is optimizing, then uses RL to learn a policy from that inferred reward. This is used when specifying a precise reward function is difficult.

The credit assignment problem is the challenge of determining which actions in a sequence are responsible for a received reward, especially when feedback is delayed or sparse. A poorly shaped reward function exacerbates this problem. RL algorithms must solve credit assignment to learn effective policies, using mechanisms like temporal-difference learning and eligibility traces to propagate rewards back to causative states and actions.

Reward shaping is the technique of designing additional intermediate rewards to guide an agent toward a sparse primary goal, making the reward function easier to learn from. For example, giving small positive rewards for moving closer to a target. It must be done carefully to avoid creating reward hacking, where the agent exploits the shaped rewards without achieving the true objective. Potential-based shaping offers theoretical guarantees against policy distortion.

This fundamental dilemma in RL involves balancing:

• Exploration: Trying new actions to gather information about their rewards and improve the policy.
• Exploitation: Choosing actions known to yield high reward based on current knowledge. The structure of the reward function directly influences this trade-off. Sparse rewards require extensive exploration, while dense, informative rewards can allow faster convergence to exploitation. Algorithms use strategies like ε-greedy, entropy regularization (as in Soft Actor-Critic), or optimism under uncertainty to manage this balance.