Policy Gradient

Policy gradient methods explicitly represent the policy as a parameterized function (e.g., a neural network), denoted as πθ(a|s). This function outputs a probability distribution over actions given a state. Optimization directly adjusts the parameters θ to increase the probability of high-reward actions. This is a key distinction from value-based methods, which learn a value function and derive a policy indirectly (e.g., via ε-greedy).

• Example: A continuous control policy for a robotic arm might output the mean and variance of a Gaussian distribution for joint torque commands.
• Advantage: Enables learning stochastic policies and naturally handles continuous action spaces.

The core update mechanism is gradient ascent on the performance objective J(θ), which is the expected cumulative reward. The fundamental update rule is derived from the policy gradient theorem:

θ ← θ + α * ∇θ J(θ)

Where α is the learning rate. The gradient ∇θ J(θ) is estimated from sampled trajectories. Unlike supervised learning, the "correct" action is not known; the gradient points in the direction that increases the probability of actions that led to higher returns. Algorithms differ primarily in how they estimate this gradient efficiently and with low variance.

Classic vanilla policy gradient methods (e.g., REINFORCE) are inherently on-policy. This means the data used to compute the gradient must be generated by the current policy πθ being optimized. After each parameter update, old data becomes obsolete because it was generated by a different policy. This leads to lower sample efficiency compared to off-policy methods, as new interactions with the environment are constantly required.

• Consequence: These methods can be computationally expensive for real-world robotics where physical interaction is slow.
• Modern variants like PPO and TRPO are also on-policy but use advanced techniques to make multiple updates from a batch of data.

A major challenge is the high variance in the Monte Carlo estimate of the policy gradient. The reward signal can be noisy and sparse, especially in long-horizon tasks. High variance leads to unstable training and slow convergence.

Key techniques to reduce variance include:

• Introducing a baseline: Subtracting a state-dependent baseline (like a value function V(s)) from the returns to create an advantage function A(s,a). This centers the updates without introducing bias.
• Actor-Critic architectures: Using a critic network to estimate the value or advantage function, providing a lower-variance target for the actor (policy) update.
• Temporal discounting (γ): Reducing the weight of future rewards in the return calculation.

A naive gradient ascent step can lead to a destructively large policy update that collapses performance. Modern algorithms enforce stability by constraining how much the policy can change per update.

• Trust Region Policy Optimization (TRPO): Uses a complex second-order optimization to enforce a hard Kullback–Leibler (KL) divergence constraint between the old and new policy.
• Proximal Policy Optimization (PPO): A simpler, first-order approximation that uses a clipped surrogate objective to penalize updates that would move the policy too far. PPO's clipped objective is a primary reason for its widespread adoption in RL practice and research.

In visuomotor control, the policy πθ is often a deep neural network that maps high-dimensional visual observations (e.g., camera pixels) directly to low-level motor commands (torques, velocities). Policy gradients are a primary method for training such end-to-end policies, especially when combined with sim-to-real transfer techniques like domain randomization.

• Challenge: The perception-action mapping is complex and non-linear. Policy gradients must discover useful visual features (like object position) through reward signals alone or from demonstrations.
• Use Case: Training a robotic arm to grasp objects from pixels using reinforcement learning in simulation, where the policy's parameters are optimized via PPO or SAC (an off-policy actor-critic method).

Unlike value-based methods that learn a value function and derive a policy, policy gradient algorithms directly parameterize and optimize the policy π(a|s; θ). They adjust the parameters θ in the direction that increases the probability of actions leading to higher long-term reward. This is achieved by estimating the gradient of the expected return J(θ) with respect to θ, often using the REINFORCE or likelihood ratio trick, which allows gradient estimation from sampled trajectories without requiring a differentiable model of the environment.

Policy gradients are the dominant approach for training continuous control policies for robotic manipulation and locomotion. Their ability to handle high-dimensional, continuous action spaces (e.g., joint torques) makes them ideal for tasks like:

• Dexterous manipulation: Learning to grasp and manipulate objects with multi-fingered hands.
• Legged locomotion: Training bipedal and quadrupedal robots to walk, run, and recover from pushes.
• Drone flight: Controlling throttle and attitude for agile, stable flight. Algorithms like PPO and SAC provide the stability and sample efficiency required for these complex, real-world control problems.

Policy gradients enable end-to-end training of visuomotor policies that map raw pixel inputs from cameras directly to low-level motor commands. This bypasses the need for hand-engineered perception pipelines. Key techniques include:

• Using convolutional neural networks as the policy's visual encoder.
• Domain randomization in simulation to bridge the sim-to-real gap.
• Auxiliary losses to encourage useful visual feature learning. This approach is fundamental for creating robots that can perform tasks like bin picking, peg insertion, and kitchen manipulation directly from visual observation.

PPO is the most widely adopted policy gradient algorithm in modern robotics and AI. It introduces a clipped surrogate objective to prevent destructively large policy updates, ensuring stable and reliable training. Its key features are:

• Clipped Probability Ratio: Constrains the policy change to a trust region.
• Multiple Epochs of Minibatch Updates: Reuses sampled data for improved sample efficiency.
• Simplicity and Robustness: Fewer hyperparameters to tune than prior methods like TRPO. PPO's balance of performance, simplicity, and stability has made it the default algorithm for training policies in simulation for real-world transfer.

Policy gradient methods naturally employ stochastic policies, which output a probability distribution over actions. This inherent randomness is a powerful mechanism for exploration. The policy's entropy can be tuned:

• High entropy encourages exploration of new actions.
• Low entropy leads to more deterministic, exploitative behavior. Algorithms like Soft Actor-Critic (SAC) explicitly maximize entropy alongside reward, promoting robust exploration and learning of diverse skills. This is critical for discovering complex behaviors in sparse-reward environments.

Policy gradients are not solely model-free. They integrate with model-based components for improved data efficiency and generalization:

• Learned Dynamics Models: A policy can be trained using gradients backpropagated through a learned forward model (model-based policy optimization).
• Model Predictive Control (MPC): A policy can provide a warm-start or prior for an online MPC planner.
• Imagination-Based Training: Policies can be trained on rollouts from a learned world model, reducing expensive real-world interaction. This hybrid approach combines the sample efficiency of model-based learning with the asymptotic performance of model-free policy optimization.

A foundational reinforcement learning architecture that combines two neural networks: an Actor (the policy) that selects actions, and a Critic (the value function) that evaluates those actions by estimating expected future reward. This structure reduces the high variance of pure policy gradients by using the critic's feedback to guide the actor's updates. Most modern policy gradient algorithms, like PPO and SAC, are actor-critic methods.

A dominant policy gradient algorithm designed for stable, data-efficient training. PPO introduces a clipped surrogate objective function that prevents destructively large policy updates. Its key mechanisms include:

• Clipped Probability Ratio: Constrains the change between the new and old policy.
• Multiple Epochs of Minibatch Updates: Reuses collected experience for better sample efficiency.
• Value Function and Entropy Losses: Often combined with the policy loss for improved performance. PPO is a workhorse for training visuomotor policies due to its robustness and relative simplicity.

An off-policy, maximum entropy reinforcement learning algorithm. SAC optimizes a policy to maximize both expected reward and entropy, encouraging exploration and learning robust, multi-modal behavior. Key features:

• Off-Policy Learning: Uses a replay buffer for improved sample efficiency.
• Automatic Temperature Tuning: Adjusts the entropy term dynamically.
• Stochastic Policy: Outputs a mean and variance for actions in continuous spaces. It is particularly effective for complex, continuous control tasks common in robotics.

The broader reinforcement learning paradigm to which policy gradients belong. In model-free RL, an agent learns a policy or value function directly from interaction with the environment, without explicitly learning or using a model of the environment's dynamics. Policy gradient methods are a primary approach within model-free RL for learning parameterized policies. This contrasts with model-based RL, which first learns a dynamics model for planning.

The formal mathematical framework for decision-making under uncertainty where an agent cannot directly observe the true state of the environment. The agent receives only partial or noisy observations. This is the standard model for visuomotor control, as a robot's camera provides a high-dimensional observation (pixels) of an underlying, unobserved physical state. Policy gradient methods can be applied to POMDPs by training recurrent policies (e.g., with LSTMs) that maintain an internal belief state over time.

A precursor to PPO that more rigorously enforces a trust region constraint during policy updates. TRPO uses a second-order optimization method (conjugate gradient) to ensure the new policy's performance does not degrade by more than a specified bound, measured by the KL-divergence between the old and new policies. While theoretically sound, its computational complexity led to the development of the simpler, first-order PPO algorithm, which approximates the trust region with clipping.