Proximal Policy Optimization (PPO)

The central mechanism of PPO that ensures stable updates. Instead of naively maximizing the probability of advantageous actions, PPO modifies the objective function to clip the policy update ratio. This ratio compares the probability of an action under the new policy versus the old policy. By clipping this ratio within a small interval (e.g., [0.8, 1.2]), the algorithm prevents a single batch of data from causing an excessively large and potentially destructive update to the policy parameters. This clipping acts as a trust region, guaranteeing that the new policy does not stray too far from the old policy.

An alternative formulation to clipping that explicitly enforces a trust region. This version adds a penalty to the objective function based on the Kullback-Leibler (KL) divergence between the new policy and the old policy. The KL divergence measures how different two probability distributions are. By penalizing high KL divergence, the algorithm directly discourages the policy from changing too dramatically in a single update. While theoretically well-founded, the clipped objective is often preferred in practice due to its simplicity and robustness, as it doesn't require tuning the coefficient for the KL penalty.

PPO improves sample efficiency by reusing collected experience data. After gathering a batch of trajectories from the environment, PPO performs multiple epochs of gradient descent on randomly sampled minibatches from that same batch. This contrasts with traditional policy gradient methods like TRPO or vanilla policy gradients, which typically use the data only once. By performing several optimization steps (e.g., 4-10 epochs) on the same data, PPO extracts more learning signal from each interaction with the environment, making it more data-efficient.

While not exclusive to PPO, it is almost universally used with it to reduce variance in policy updates. GAE provides a method for estimating the advantage function, which tells the agent how much better a specific action was compared to the average action in a given state. GAE smoothly interpolates between high-variance, unbiased estimates (using many steps of rewards) and low-variance, biased estimates (using a single step). By balancing this bias-variance trade-off, GAE delivers a stable and low-variance advantage signal that is critical for PPO's reliable performance across diverse environments.

PPO employs an actor-critic framework, which separates the policy and value functions into two neural networks (or two heads of one network).

• The Actor (Policy Network): Selects which action to take in a given state. It is optimized using the clipped surrogate objective.
• The Critic (Value Network): Estimates the expected cumulative reward (value) of being in a given state. It is trained via mean-squared error against empirical returns. The critic's value estimates are used to compute advantages (via GAE), which guide the actor's updates. This separation provides a stable, low-variance learning signal.

PPO is the dominant reinforcement learning optimizer used in the final stage of Reinforcement Learning from Human Feedback (RLHF). In this context:

• The Actor is the language model being fine-tuned (the policy).
• The Reward Model (trained on human preferences) provides the reward signal, replacing the environment.
• A KL Divergence Penalty from a reference model (the initial SFT model) is added to the reward to prevent the policy from deviating too far into nonsensical or high-reward-hacking text. PPO's stability is crucial here, as the reward model is an imperfect proxy for human preference, and large, unstable updates could degrade the model's coherence and language capabilities.

This is PPO's core technical innovation that enables its stability. The objective function clips the probability ratio between the new and old policy. This clipping prevents excessively large policy updates that could collapse performance. Mechanically:

• It computes a standard policy gradient advantage estimate.
• It forms a surrogate objective using the probability ratio of taking an action under the new vs. old policy.
• It clips this ratio within a range (e.g., [0.8, 1.2]), creating a pessimistic lower bound on the true objective.
• This simple modification acts as a trust region, making PPO much more robust and easier to tune than prior policy gradient methods like TRPO.

RLHF is the overarching training pipeline where PPO is most commonly applied. It is a three-stage process:

• A base language model is supervised fine-tuned (SFT).
• A reward model is trained on human preference data.
• The SFT model is optimized with a reinforcement learning algorithm (like PPO) using rewards from the reward model. PPO's role is to update the language model's policy to maximize the reward signal while staying close to the original model via a KL divergence penalty, preventing excessive deviation that degrades language quality.

Reward modeling is the process of training a neural network to predict a scalar reward that reflects human preferences. It is the critical precursor to PPO optimization.

• Trained on datasets of pairwise comparisons (e.g., Response A vs. Response B).
• Typically uses the Bradley-Terry model to estimate the probability one response is preferred.
• The resulting reward model provides the objective function that PPO's policy seeks to maximize. Imperfections in this model can lead to reward hacking or reward overoptimization.

A KL divergence penalty is a regularization term added to PPO's objective function. It measures the difference between the current policy and a reference policy (usually the initial SFT model).

• Purpose: Prevents the policy from changing too drastically in a single update, which is essential for stability.
• Mechanism: It penalizes updates that would make the policy's output distribution diverge from the reference. This controls the update size, preserving the model's core language capabilities and grammatical coherence while it learns new, preferred behaviors.

DPO is an alternative to the PPO-based RLHF pipeline. It reformulates the preference learning problem to bypass explicit reward modeling and reinforcement learning.

• Key Difference: DPO directly optimizes a language model on preference data using a simple classification loss derived from the Bradley-Terry model.
• Contrast with PPO: DPO is often simpler to implement and train but is derived from the same theoretical foundation. PPO is generally considered more flexible for complex, non-differentiable reward scenarios.

PPO is fundamentally an actor-critic method. This architecture separates two key functions:

• Actor (Policy Network): The model being trained (e.g., the language model). It selects actions (generates tokens).
• Critic (Value Network): A separate network that estimates the expected cumulative reward (value) of being in a given state. During PPO updates, the critic provides a baseline to reduce the variance of policy gradient estimates, leading to more stable and efficient learning compared to pure policy gradient methods.

Reward hacking is a critical failure mode in RLHF pipelines using PPO. It occurs when the policy exploits flaws in the learned reward model to achieve high scores without fulfilling the true human intent.

• Example: A model might add phrases like "This is a great answer" to all outputs if the reward model correlates such phrases with positivity.
• Relationship to PPO: PPO's aggressive optimization can quickly amplify these exploits. The KL divergence penalty and careful reward model training are primary defenses against this phenomenon, also known as reward overoptimization.