Policy Gradient

The foundational policy gradient algorithm. It uses a Monte Carlo estimate of the return from complete trajectories to compute the gradient of expected reward.

• Mechanism: Updates policy parameters in the direction that increases the probability of actions that led to high total reward.
• Key Feature: It is an on-policy and high-variance method, as it relies on full episode returns.
• Use Case: Often serves as a conceptual starting point; practical applications use its variance-reduced derivatives.

A hybrid architecture that combines a policy network (Actor) with a value network (Critic). The critic reduces variance by providing a baseline.

• Mechanism: The actor proposes actions, while the critic evaluates the state or state-action value, guiding the actor's updates.
• Key Feature: Significantly lower variance than pure REINFORCE, leading to more stable and sample-efficient learning.
• Foundation: Serves as the basis for most advanced policy gradient algorithms like A3C, PPO, and SAC.

A dominant, robust policy gradient algorithm that uses a clipped surrogate objective to constrain policy updates.

• Mechanism: Prevents destructively large policy updates by clipping the probability ratio between new and old policies.
• Key Feature: Reliable performance with simple tuning, making it a default choice for complex environments including robotic simulations.
• Practical Impact: Extensively used in sim-to-real pipelines due to its stability and ease of parallelization.

The theoretical precursor to PPO. It rigorously enforces a trust region constraint using conjugate gradient and Fisher information matrix approximations.

• Mechanism: Maximizes a surrogate objective subject to a KL-divergence constraint on the policy change per step.
• Key Feature: Guarantees monotonic improvement under theory, but is computationally complex.
• Role: Provides the mathematical justification for PPO's clipped objective, which approximates the trust region more efficiently.

An off-policy, maximum entropy actor-critic algorithm designed for continuous control.

• Mechanism: Maximizes both expected reward and policy entropy, encouraging exploration and robustness.
• Key Feature: State-of-the-art sample efficiency and robustness on continuous control benchmarks, making it ideal for robotic manipulation and locomotion.
• Advantage: Being off-policy, it can reuse past experience from a replay buffer, which is critical when real-world robot data is expensive.

An off-policy actor-critic algorithm that extends DQN to continuous action spaces using a deterministic policy.

• Mechanism: Employs a replay buffer and target networks for stability, applying the deterministic policy gradient theorem.
• Key Feature: One of the first successful deep RL algorithms for continuous control (e.g., robotic arm tasks).
• Consideration: Can be less stable and more sensitive to hyperparameters than PPO or SAC, but is foundational.

Actor-Critic is a foundational architecture that combines the strengths of policy-based and value-based methods. It uses two networks: an actor (the policy) that selects actions, and a critic (a value function) that evaluates those actions. The critic provides a lower-variance estimate of the advantage function, which is used to update the actor. This separation leads to more stable and sample-efficient learning compared to pure policy gradient methods like REINFORCE. Common algorithms like A3C, A2C, and PPO are built on the actor-critic framework.

Proximal Policy Optimization (PPO) is a dominant policy gradient algorithm designed for stability and ease of use. It addresses a core challenge in policy gradients: ensuring that policy updates do not collapse performance. PPO introduces a clipped surrogate objective that prevents the new policy from deviating too far from the old policy. Key features include:

• Clipped Probability Ratios: Limits the magnitude of policy updates.
• Multiple Epochs: Reuses collected data for several gradient steps.
• Generalized Advantage Estimation (GAE): Often used for low-variance advantage estimates. PPO is the default algorithm for many continuous control benchmarks and sim-to-real transfer projects due to its reliable performance.

Soft Actor-Critic (SAC) is an off-policy, maximum entropy actor-critic algorithm optimized for continuous control tasks. Its core innovation is modifying the standard RL objective to maximize both expected reward and the entropy of the policy. This encourages exploration and results in more robust policies that can handle multiple successful strategies. SAC maintains:

• A stochastic actor (policy) network.
• Two critic (Q-function) networks to mitigate overestimation bias.
• A temperature parameter that automatically adjusts the trade-off between reward and entropy. Its off-policy nature, using a replay buffer, makes it highly sample-efficient, a critical property for real-world robotic learning where data is costly.

Deep Deterministic Policy Gradient (DDPG) is an off-policy, actor-critic algorithm designed for continuous action spaces. It combines insights from DQN (Deep Q-Network) with a deterministic policy gradient. Key components include:

• A deterministic actor that maps states directly to precise actions.
• A critic that estimates the Q-value of state-action pairs.
• Replay Buffer for storing and sampling past experiences.
• Target Networks (for both actor and critic) to stabilize training by providing fixed regression targets. While foundational, DDPG can be sensitive to hyperparameters. Later algorithms like TD3 (Twin Delayed DDPG) address its tendency to overestimate Q-values.

REINFORCE, or the Monte Carlo policy gradient, is the simplest policy gradient algorithm. It represents the foundational score function estimator. The policy parameters are updated in the direction that increases the probability of actions that led to high cumulative reward (return). Its update rule is: ∇J(θ) ≈ E[ G_t * ∇ log π(a|s; θ) ].

• Monte Carlo: Uses complete episode returns (G_t), leading to high variance but unbiased estimates.
• No Critic: It is a pure policy gradient method without a value function baseline, though adding one reduces variance. While rarely used alone in complex robotics due to high sample inefficiency, understanding REINFORCE is essential for grasping more advanced policy gradient derivations.

The Natural Policy Gradient and its practical implementation, Trust Region Policy Optimization (TRPO), address a fundamental issue: how large a policy update step is safe. Instead of using the standard gradient in parameter space, the natural gradient uses the Fisher Information Matrix to perform updates in the space of probability distributions. This results in updates that maximize reward while enforcing a constraint on the KL-divergence between the old and new policies.

• TRPO approximates this with a conjugate gradient solver and a line search.
• It provides monotonic improvement guarantees but is computationally complex. PPO was later developed as a simpler, first-order approximation to TRPO's constrained optimization, achieving similar performance with greater implementation ease.