Policy Gradient

Unlike value-based methods that learn a value function and derive a policy indirectly, policy gradient algorithms directly parameterize and optimize the policy itself. The policy, typically a neural network with parameters θ, outputs a probability distribution over actions given a state (π_θ(a|s)). The gradient of the expected reward with respect to θ is then ascended to improve performance.

• Key Advantage: Naturally handles continuous action spaces and stochastic policies.
• Example: A robotic arm's policy network outputs the mean and variance for a Gaussian distribution over joint torque values.

The core update mechanism is gradient ascent on a performance objective J(θ), which is the expected cumulative reward. The fundamental policy gradient theorem provides the formula for the unbiased gradient estimate: ∇_θ J(θ) = E[∇_θ log π_θ(a|s) * G_t], where G_t is the return (cumulative future reward).

• Mechanism: High-return trajectories have their action probabilities increased; low-return trajectories have them decreased.
• On-Policy Learning: The expectation is taken under the current policy, making most vanilla policy gradient methods on-policy. They require fresh samples after each update.

A primary challenge is the high variance of the gradient estimate. Since the return G_t can vary widely across trajectories, the updates are noisy, leading to unstable training. This is intrinsically linked to the credit assignment problem—determining which actions in a long sequence were responsible for the final outcome.

• Mitigation Techniques: Algorithms employ baselines (like a value function) to reduce variance without introducing bias. The Advantage function (A(s,a) = Q(s,a) - V(s)) is a common, effective baseline.
• Consequence: Requires careful engineering (e.g., advantage estimation, reward scaling) and often more samples than model-based methods.

Policy gradient methods inherently learn stochastic policies, which provide natural exploration. The policy network outputs probabilities, ensuring the agent continuously samples different actions from the distribution. This contrasts with value-based methods that often require explicit exploration heuristics (like ε-greedy) added to a deterministic greedy policy.

• Benefit: Can find optimal stochastic policies (e.g., in non-transitive games like Rock-Paper-Scissors).
• Trade-off: The level of exploration is automatically tuned by the policy's entropy, but may require entropy regularization to prevent premature convergence to a suboptimal deterministic policy.

Most advanced policy gradient methods use an actor-critic architecture. This hybrid framework combines the strengths of both policy-based and value-based methods:

• Actor: The policy network (π_θ) that selects actions.
• Critic: A value network (V_φ) that estimates the value of states or state-action pairs.

The critic evaluates the actor's actions by calculating the advantage function, which is then used to update the actor. This provides a low-variance baseline for the policy gradient.

• Examples: Advantage Actor-Critic (A2C), Asynchronous Advantage Actor-Critic (A3C), and Soft Actor-Critic (SAC) for continuous control all utilize this architecture.

A hybrid reinforcement learning architecture that combines a policy network (actor) and a value network (critic). The actor selects actions, while the critic evaluates the chosen action by estimating the value function. The critic's evaluation provides a lower-variance, bootstrapped feedback signal (the advantage) to update the actor's policy, making training more stable than pure policy gradient methods.

• Actor: Directly parameterizes the policy π(a|s; θ).
• Critic: Estimates the value function V(s; w) or Q(s,a; w).
• Advantage Function: A(s,a) = Q(s,a) - V(s) is often used as the critic's feedback to the actor.

A specific, dominant policy gradient algorithm designed for stable and sample-efficient training. PPO introduces a clipped surrogate objective that prevents destructively large policy updates. It optimizes a first-order approximation of the policy performance while constraining the change in the policy distribution (measured by KL divergence) between updates.

• Clipped Objective: Maximizes a modified reward function that penalizes policy changes beyond a trusted region.
• Trust Region: Ensures the new policy does not deviate too far from the old policy, preventing collapse.
• Empirical Success: The default algorithm for many complex environments (e.g., robotic control, game AI) due to its robustness.

A foundational Monte Carlo policy gradient algorithm. It directly implements the policy gradient theorem by using complete episode returns to estimate the gradient. The update increases the probability of actions that led to high total reward and decreases the probability of actions that led to low reward.

• Monte Carlo: Requires completing an entire episode before performing an update.
• High Variance: The gradient estimate relies on a single sampled trajectory, leading to noisy updates.
• Credit Assignment: Struggles with assigning credit to individual actions in long sequences, a problem addressed by later methods like actor-critic.

The mathematical foundation for all policy gradient methods. This theorem provides an analytical expression for the gradient of the performance measure J(θ) (expected return) with respect to the policy parameters θ. Crucially, the gradient does not require the derivative of the state distribution, which is unknown.

• Core Equation: ∇θ J(θ) ∝ Eπ [Qπ(s,a) ∇θ log π(a|s; θ)]
• Score Function: The term ∇θ log π(a|s; θ) is the score function or likelihood ratio.
• Reduces to REINFORCE: When the Q-value is replaced with the empirical return Gt.

A fundamental dichotomy in RL defining how data is used for learning. Policy gradient methods are typically on-policy.

• On-Policy (e.g., REINFORCE, PPO): The agent learns from experience collected using the current policy being improved. Data must be freshly generated after each update.
• Off-Policy (e.g., Q-Learning, DDPG): The agent can learn from experience generated by an older behavior policy, enabling reuse of past data via experience replay.
• Trade-off: On-policy methods are often more stable but less sample-efficient. Policy gradient algorithms like PPO are designed to maximize on-policy sample efficiency.

The core dilemma an RL agent faces: choosing between exploring new actions to gather information and exploiting known actions to maximize reward. Policy gradient methods handle this intrinsically through the policy's stochasticity.

• Stochastic Policy: A policy π(a|s) outputs a probability distribution over actions. The entropy of this distribution controls exploration.
• Entropy Bonus: Algorithms like Soft Actor-Critic (SAC) explicitly add an entropy term to the reward, encouraging the policy to remain stochastic and explore.
• Convergence: As learning progresses, a well-tuned policy gradient method will naturally reduce entropy, converging to a deterministic, exploitative policy for optimal actions.