Policy Gradient

Policy gradient methods directly optimize the policy π(a|s; θ) by adjusting parameters θ to maximize expected cumulative reward. This eliminates the need to derive a policy from a value function, making them naturally suited for continuous action spaces where value-based methods struggle with the maximization step.

• Policy is represented as a parameterized function (e.g., neural network)
• Outputs a probability distribution over actions rather than a single deterministic choice
• Enables learning stochastic policies that are essential in partially observable environments
• Common parameterizations include Gaussian policies for continuous control and softmax for discrete actions

The policy gradient theorem provides the mathematical foundation by expressing the gradient of expected reward as an expectation over state-action trajectories. It eliminates the need to differentiate through the environment dynamics, making model-free optimization tractable.

• Gradient formula: ∇J(θ) = E[∇log π(a|s; θ) · Q(s,a)]
• The score function ∇log π(a|s; θ) indicates how to adjust probabilities
• Does not require knowledge of transition probabilities or reward functions
• Forms the basis for algorithms like REINFORCE, PPO, and TRPO

Raw policy gradient estimates suffer from high variance, making learning unstable. The advantage function A(s,a) = Q(s,a) - V(s) measures how much better an action is compared to the average, dramatically reducing variance without introducing bias.

• Subtracting the state-value baseline V(s) centers the gradient estimate
• Advantage can be estimated using Generalized Advantage Estimation (GAE) for smooth bias-variance tradeoffs
• Critical for scaling policy gradients to complex problems with long horizons
• Enables the Actor-Critic architecture where the critic estimates the advantage for the actor

Modern policy gradient methods constrain updates to a trust region to prevent catastrophic policy collapse from overly large parameter changes. TRPO enforces a hard KL-divergence constraint, while PPO uses a simpler clipped surrogate objective that achieves similar stability with less computational overhead.

• Prevents the new policy from diverging too far from the old policy
• PPO's clipped objective: min(r(θ)·A, clip(r(θ), 1-ε, 1+ε)·A)
• Enables monotonic improvement guarantees under certain conditions
• PPO has become the default algorithm for many applications due to its robustness and ease of implementation

Policy gradient methods naturally maintain exploration through the entropy of the learned stochastic policy. By maximizing an entropy bonus alongside the reward, algorithms like Soft Actor-Critic (SAC) encourage the policy to remain exploratory, preventing premature convergence to suboptimal deterministic strategies.

• Entropy regularization: J(θ) = E[r + α·H(π(·|s))]
• Automatically balances exploration and exploitation without explicit ε-greedy schedules
• Essential for multi-modal reward landscapes where multiple good strategies exist
• Temperature parameter α controls the exploration-exploitation tradeoff

Policy gradient methods integrate seamlessly with value function approximation in the Actor-Critic framework. The actor (policy) selects actions while the critic (value function) evaluates them, combining the strengths of both policy-based and value-based approaches.

• Actor updates: uses policy gradient with critic-provided advantage estimates
• Critic updates: uses temporal difference learning or Monte Carlo returns
• Enables bootstrapping for more sample-efficient learning than pure Monte Carlo methods
• Foundational architecture behind A3C, A2C, PPO, and SAC

A hybrid architecture that combines the policy-based actor with a value-based critic. The actor (parameterized by θ) selects actions using the policy gradient, while the critic estimates the value function to reduce variance in the gradient estimate.

• Actor: Learns policy π(a|s) directly
• Critic: Estimates V(s) or Q(s,a) to provide a baseline
• Advantage: A(s,a) = Q(s,a) - V(s) quantifies how much better an action is than average
• Reduces the high variance of pure Monte Carlo policy gradients

Common implementations include A2C (synchronous) and A3C (asynchronous), which parallelize training across multiple environment instances.

A state-of-the-art policy gradient algorithm developed by OpenAI that constrains policy updates to a trust region to prevent destructively large parameter changes. PPO uses a clipped surrogate objective function.

• Clipping mechanism: Limits the probability ratio r(θ) between old and new policies to [1-ε, 1+ε]
• Surrogate objective: L(θ) = min(r(θ)Â, clip(r(θ), 1-ε, 1+ε)Â)
• Avoids the computational complexity of TRPO's second-order optimization
• Balances sample efficiency with implementation simplicity

PPO has become the default algorithm for continuous control tasks and is widely used in production RL systems due to its robustness across hyperparameters.

A function that quantifies the relative benefit of taking a specific action compared to the average action in a given state. It is central to reducing variance in policy gradient estimates.

• Definition: A(s,a) = Q(s,a) - V(s)
• Interpretation: Positive advantage means the action is better than average; negative means worse
• Generalized Advantage Estimation (GAE): Balances bias and variance using a λ-weighted sum of n-step temporal difference errors
• Replaces raw returns in the policy gradient: ∇J(θ) = E[∇log π(a|s) · A(s,a)]

Using the advantage function instead of raw Q-values centers the gradient estimate, dramatically reducing variance without introducing bias.

The foundational Monte Carlo policy gradient algorithm that estimates the gradient of expected reward by sampling complete trajectories and weighting log-probabilities by total discounted return.

• Update rule: θ ← θ + α · G_t · ∇log π(a_t|s_t)
• G_t: Total discounted return from timestep t to end of episode
• Unbiased but high variance: Requires many samples for stable learning
• Episodic only: Must wait until episode completion for the return

REINFORCE is the simplest policy gradient method and serves as the pedagogical starting point. Modern variants add baselines (REINFORCE with baseline) to reduce variance, forming the conceptual bridge to actor-critic methods.

The mathematical framework that formalizes the environment for policy gradient methods. An MDP is defined by the tuple (S, A, P, R, γ) where the agent interacts sequentially to maximize cumulative reward.

• S: State space representing all possible situations
• A: Action space of available decisions
• P(s'|s,a): Transition probability to next state s'
• R(s,a): Immediate reward function
• γ ∈ [0,1]: Discount factor balancing immediate vs. future rewards

Policy gradient methods directly search the policy space π(a|s) without requiring knowledge of the transition model P, making them model-free and applicable to complex real-world environments where dynamics are unknown.

The fundamental dilemma in policy optimization between gathering new information about uncertain state-action pairs and leveraging known high-reward behaviors. Policy gradient methods handle this through stochastic policies.

• Stochastic policies: Output a probability distribution over actions, naturally exploring
• Entropy regularization: Adds H(π(·|s)) to the objective to encourage exploration
• Entropy coefficient: Controls the exploration-exploitation balance; decays over time
• Deterministic policy gradient (DPG): Uses off-policy exploration with a separate behavior policy

Unlike value-based methods that use ε-greedy exploration, policy gradient methods maintain a smooth, differentiable exploration mechanism through the policy's probability distribution.