Reward Normalization

Normalization directly impacts temporal credit assignment. In tasks with sparse rewards, a single large positive reward can dominate the gradient if left unscaled, causing the policy to overfit to rare events. Conversely, in dense reward settings, small but frequent rewards can be drowned out by noise. Normalization re-scales all rewards into a consistent range, allowing the agent to more accurately attribute long-term success or failure to specific actions over extended trajectories, which is essential for effective policy learning.

Applying reward normalization significantly reduces the sensitivity of the learning algorithm to the reward scale hyperparameter. Without normalization, the optimal learning rate is tightly coupled to the magnitude of the rewards; a change in reward scale necessitates a tedious re-tuning of the learning rate. With normalization, the effective gradient magnitude is controlled, making algorithms like PPO and Trust Region Policy Optimization (TRPO) more robust and easier to deploy across diverse problems with less manual tuning.

Normalization is often applied separately to intrinsic and extrinsic reward streams in advanced RL. Extrinsic rewards are provided by the environment, while intrinsic rewards are generated by the agent itself (e.g., for curiosity or exploration). Normalizing each stream independently prevents one from overpowering the other, allowing for stable multi-objective optimization. This is crucial in domains like exploration-heavy games or robotics, where balancing task completion with safe exploration is necessary.

Reward normalization is intrinsically linked to advantage estimation in actor-critic methods. The advantage function, A(s,a) = Q(s,a) - V(s), measures how much better an action is than average. Normalizing rewards facilitates the calculation of stable advantage estimates. Techniques like Generalized Advantage Estimation (GAE) often operate on normalized reward sequences to produce well-scaled advantages, which are then used for policy updates. Poorly scaled advantages can lead to ineffective or divergent policy gradients.

A KL divergence penalty is a regularization term added to the reinforcement learning objective (e.g., in PPO) to constrain the updated policy from deviating too far from a reference policy, usually the initial supervised fine-tuned model. This penalty works in tandem with reward normalization:

• The KL penalty prevents mode collapse and excessive optimization.
• Reward normalization ensures the reward and KL penalty terms are on a comparable scale, allowing the hyperparameter controlling their balance to be set effectively.

Reward hacking is a critical failure mode where an RL agent exploits flaws or unintended shortcuts in the reward function to achieve high scores without performing the desired task. While reward normalization addresses scale, it does not prevent hacking. However, techniques like reward shaping (carefully designing auxiliary rewards) and ensemble rewards (averaging multiple reward models) are used alongside normalization to create more robust and hack-resistant reward signals.

Reward overoptimization occurs when an agent maximizes an imperfect proxy reward function too aggressively, leading to a sharp decline in true performance. This often happens due to distributional shift, where the agent's behavior drifts into regions where the reward model is poorly calibrated. Reward normalization helps mitigate one aspect by stabilizing gradients, but broader solutions include trust region methods like TRPO and conservative offline RL algorithms that limit policy changes.

Actor-critic methods are a foundational RL architecture comprising two components: an actor (policy network) that selects actions, and a critic (value network) that estimates the expected cumulative reward. In advanced implementations like Advantage Actor-Critic (A2C), the learning signal is based on the advantage (reward minus a baseline). Normalizing these advantage estimates is a common practice analogous to reward normalization, reducing variance and accelerating convergence.