Policy Gradient Methods

Unlike value-based methods that learn a value function and derive a policy, policy gradient methods directly parameterize the policy π(a|s; θ) as a function (e.g., a neural network) with parameters θ. This allows them to naturally handle:

• Continuous action spaces without requiring a maximization over actions.
• Learning stochastic policies, which are essential for exploration and tasks requiring random behavior.
• Representing complex, high-dimensional policies where the optimal action selection is a complex function of the state.

The core update mechanism is gradient ascent on the performance objective J(θ), which is typically the expected cumulative reward. The fundamental policy gradient theorem provides the gradient formula: ∇_θ J(θ) = E[∇_θ log π(a|s; θ) * Q^π(s, a)] This equation shows that the update pushes the policy parameters in the direction that increases the probability of actions (log π) that lead to high state-action values (Q^π). The expectation is approximated by sampling trajectories from the environment, making it a Monte Carlo method.

A primary challenge is the high variance of the Monte Carlo gradient estimates. Since updates are based on full episode returns, the signal can be noisy, leading to unstable learning. Key techniques to reduce variance include:

• Introducing a baseline, most commonly a state-value function V(s), transforming the update to use the advantage function A(s,a) = Q(s,a) - V(s).
• Actor-Critic architectures, where a 'critic' network estimates V(s) or A(s,a) to provide a lower-variance learning signal for the 'actor' policy.
• Temporal Difference (TD) methods to bootstrap value estimates, as used in many modern algorithms like PPO and SAC.

The standard policy gradient theorem assumes that the data used for the update is generated by the current policy being optimized. This makes vanilla policy gradient methods on-policy. Consequences include:

• Sample inefficiency: Data from old policies cannot be reused, requiring constant fresh interaction with the environment.
• This limitation spurred the development of advanced off-policy policy gradient methods (like DDPG, SAC) and importance sampling techniques to reuse past data.
• Algorithms like PPO use trust region or clipped objective methods to allow multiple epochs of updates on a batch of data while approximating the on-policy requirement.

Policy gradient methods are the dominant approach for reinforcement learning in continuous control tasks, such as robotic manipulation and locomotion. This is due to:

• The natural parameterization of a policy that outputs continuous action vectors (e.g., joint torques).
• Algorithms like Deep Deterministic Policy Gradient (DDPG), Proximal Policy Optimization (PPO), and Soft Actor-Critic (SAC) have become standard benchmarks for continuous control benchmarks (e.g., OpenAI Gym's MuJoCo, DeepMind Control Suite).
• Their ability to learn smooth, stable control policies directly from high-dimensional sensory input.

Policy gradient methods are a form of gradient-based policy search. They are distinguished from gradient-free policy search methods (e.g., Evolutionary Strategies, CMA-ES) which optimize parameters by evaluating perturbations without computing a gradient. Key differentiators:

• Sample Efficiency: Gradient-based methods typically use far fewer samples by leveraging gradient direction.
• Scalability: Gradients via backpropagation scale to policies with millions of parameters.
• Hybrid Approaches: Some methods, like Evolution Strategies, can be viewed as estimating a gradient in parameter space using finite differences over a population, blurring the line between the two families.

The foundational policy gradient theorem algorithm. It is a Monte Carlo method, meaning it requires complete episode trajectories to compute returns before performing an update.

• Mechanism: Updates policy parameters in the direction that increases the probability of actions proportional to the total reward received from that state onward (the return).
• Key Feature: High variance due to Monte Carlo return estimation. Often requires a baseline (like a value function) to reduce variance and stabilize learning.
• Robotics Use Case: Suitable for episodic tasks where a clear success/failure signal is received only at the end, such as a robot completing a manipulation sequence.

A hybrid architecture that combines a policy network (Actor) with a value network (Critic). The actor selects actions, while the critic evaluates the chosen action by estimating the state-value or advantage function.

• Mechanism: The critic provides a lower-variance, bootstrapped estimate (e.g., TD error) to guide the actor's updates, replacing the high-variance Monte Carlo return used in REINFORCE.
• Key Feature: Dramatically improves sample efficiency and training stability compared to pure Monte Carlo methods.
• Robotics Use Case: Ideal for continuous control tasks (e.g., joint torque control for a robotic arm) where online, incremental learning is required.

An on-policy algorithm designed to ensure monotonic policy improvement by enforcing a trust region constraint on policy updates.

• Mechanism: Maximizes a surrogate objective function subject to a constraint on the Kullback–Leibler (KL) divergence between the old and new policies. This prevents overly large, destabilizing updates.
• Key Feature: Provides strong theoretical guarantees but is computationally complex, requiring conjugate gradient descent to approximate the natural policy gradient.
• Robotics Use Case: Effective for training stable, reliable policies on physical hardware where catastrophic policy collapse during training must be avoided.

A widely adopted on-policy algorithm that approximates TRPO's stability with a simpler, first-order optimization objective.

• Mechanism: Uses a clipped surrogate objective that penalizes policy changes which would move too far from the previous policy. This clipping acts as a soft trust region.
• Key Feature: Strikes a balance between ease of implementation, sample efficiency, and empirical performance. It is the default choice for many continuous control benchmarks.
• Robotics Use Case: The go-to algorithm for training policies in physics-based simulators (e.g., MuJoCo, Isaac Sim) prior to sim-to-real transfer.

An off-policy, actor-critic algorithm specifically designed for continuous action spaces.

• Mechanism: Employs a deterministic policy (actor) that outputs exact action values. It uses a Q-function critic (learned via off-policy TD updates) and incorporates techniques from DQN: a replay buffer and target networks for stability.
• Key Feature: Extends Q-learning to continuous domains by using the critic's gradient to train the deterministic actor.
• **Robotics Use Case": Training robotic controllers where actions are high-dimensional and continuous, such as velocity commands for a mobile base or torque vectors for a multi-joint manipulator.

An off-policy, maximum entropy actor-critic algorithm that aims to maximize both expected reward and policy entropy.

• Mechanism: The entropy term encourages exploration by favoring stochastic policies. It automatically trades off between exploring new behaviors and exploiting known rewards, leading to improved robustness and sample efficiency.
• Key Feature: Known for its state-of-the-art performance and stability on a wide range of continuous control benchmarks. It maintains a stochastic actor and uses two Q-networks with a minimum target to combat overestimation bias.
• Robotics Use Case: Excellent for learning complex, contact-rich manipulation skills and dexterous locomotion where the agent must explore a wide range of behaviors to find robust solutions.

A foundational RL framework that combines a policy network (actor) and a value network (critic). The actor selects actions, while the critic evaluates them by estimating the value function. This architecture reduces the high variance of pure policy gradients by using the critic's feedback to guide the actor's updates. Most advanced policy gradient methods, like PPO and SAC, are actor-critic algorithms.

A dominant policy gradient algorithm known for its stability and ease of implementation. PPO optimizes a clipped surrogate objective that prevents large, destabilizing policy updates. Its key features include:

• Clipped Probability Ratio: Constrains the policy change to a trust region.
• Multiple Epochs: Reuses sampled data for several gradient updates, improving sample efficiency.
• It is a workhorse for robotic simulation training due to its robustness across diverse continuous control tasks.

An off-policy, maximum entropy actor-critic algorithm designed for continuous action spaces. SAC maximizes both expected reward and policy entropy, which encourages exploration and leads to more robust policies. It is particularly effective in robotics because:

• The entropy term prevents premature convergence to sub-optimal behaviors.
• Its off-policy nature allows efficient reuse of past experience via a replay buffer.
• It automatically trades off exploration and exploitation via a learnable temperature parameter.

A policy optimization algorithm that guarantees monotonic improvement by enforcing a hard constraint on policy updates. It uses the Kullback-Leibler (KL) divergence to define a trust region within which the new policy is optimized. While mathematically sound, it is computationally complex, involving conjugate gradient methods to approximate the natural policy gradient. PPO was developed as a simpler, more practical approximation of TRPO's trust region concept.

A foundational Monte Carlo policy gradient method. It directly estimates the policy gradient using complete episode returns. The update rule adjusts policy parameters in the direction that increases the probability of actions that led to high cumulative reward. While simple, it suffers from high variance, leading to unstable learning. Modern policy gradient methods build upon REINFORCE by incorporating baselines (like a critic) to reduce this variance.

A theorem that provides the gradient for deterministic policies (which output a specific action, not a distribution). The Deep Deterministic Policy Gradient (DDPG) algorithm extends this into the deep RL domain. DDPG is an actor-critic, off-policy algorithm ideal for continuous action spaces in robotics, such as precise joint control. It combines:

• A deterministic actor network.
• A critic Q-network.
• Experience replay and target networks for stability.