Policy Gradient Methods

Unlike value-based methods (e.g., Q-Learning) that first learn a value function and then derive a policy, policy gradient algorithms directly optimize the parameters of a stochastic policy function, typically represented by a neural network. The gradient of the expected cumulative reward with respect to the policy parameters is estimated, and gradient ascent is performed to increase the probability of high-reward actions. This direct parameterization allows for learning stochastic policies and functioning in continuous action spaces where enumerating actions for a Q-function is infeasible.

The mathematical foundation is the Policy Gradient Theorem, which provides an analytical expression for the gradient of the performance objective. For the standard objective J(θ), the theorem shows the gradient is the expected value of the score function multiplied by the action's return. Crucially, the gradient does not require derivatives of the environment's state transition dynamics, making the approach model-free. The general form is: ∇J(θ) = E[∇ log π(a|s; θ) * G_t], where G_t is a return estimator like the total discounted reward.

A primary challenge is the high variance of the Monte Carlo-style gradient estimates derived from trajectory samples. Because the return G_t over an entire episode can vary significantly, the raw policy gradient suffers from noisy updates that slow convergence. This characteristic necessitates the use of variance reduction techniques, the most common being:

• Baselines: Subtracting a state-dependent baseline (like a value function estimate) from the return.
• Actor-Critic Methods: Using a critic network to estimate the advantage function A(s,a) = Q(s,a) - V(s), which directly measures how much better an action is than average.

Standard vanilla policy gradient methods (e.g., REINFORCE) are inherently on-policy. They learn the value of and optimize the policy that is actively being used to collect experience. Data from old policies cannot be reused because the gradient update is specific to the policy parameters that generated the trajectories. This leads to poor sample efficiency compared to off-policy methods. Modern algorithms like PPO and TRPO remain on-policy but improve efficiency by enabling multiple gradient steps from a batch of data through constrained updates.

Standard gradient ascent can lead to catastrophic performance drops if a step is too large, as the policy distribution changes abruptly. Trust Region methods address this by constraining how much the policy can change per update, measured by the Kullback-Leibler (KL) divergence between the old and new policy. Natural Policy Gradient and TRPO approximate this by using the Fisher information matrix to pre-condition the gradient, resulting in updates that follow the steepest ascent direction in policy space, not parameter space. PPO simplifies this with a clipped surrogate objective that empirically enforces a trust region.

To prevent premature convergence to suboptimal deterministic policies, a common technique is to add an entropy bonus to the objective function. This encourages the policy to maintain a degree of randomness, promoting exploration. The entropy of a policy π, H(π) = -Σ π(a|s) log π(a|s), is maximized when the policy is uniform. Algorithms like Soft Actor-Critic (SAC) explicitly maximize entropy alongside reward, leading to more robust exploration and improved stability. This characteristic is particularly valuable in complex environments with sparse or deceptive reward signals.

A foundational Monte Carlo policy gradient method. REINFORCE directly estimates the policy gradient using complete episode trajectories. It operates by:

1. Sampling a trajectory under the current policy.
2. Calculating the return (sum of discounted rewards) from each state.
3. Updating the policy parameters in the direction that increases the probability of actions that led to high returns. While simple and conceptually clear, REINFORCE suffers from high variance in gradient estimates because it relies on full Monte Carlo returns. This variance is often mitigated with a baseline, such as a state-value function, which transforms it into an actor-critic method.

A policy gradient theorem for deterministic policies. While standard policy gradients are derived for stochastic policies, DPG shows how to compute gradients for deterministic policies (which output a single action for a given state). This is particularly useful in continuous action spaces. The Deep Deterministic Policy Gradient (DDPG) algorithm extends this idea, combining DPG with deep neural networks and experience replay. DDPG is an off-policy actor-critic method that is effective for tasks requiring precise, continuous control but can be sensitive to hyperparameters.