Policy Gradient

The foundational policy gradient algorithm. It uses a Monte Carlo estimate of the return from complete episodes to compute the gradient for updating policy parameters.

• Core Mechanism: Updates parameters in the direction that increases the probability of actions proportional to the total reward received after taking that action.
• High Variance: As a Monte Carlo method, it suffers from high variance in gradient estimates, leading to unstable learning.
• Baseline Subtraction: A common technique to reduce variance without introducing bias involves subtracting a state-dependent baseline (like a value function estimate) from the return.

A hybrid architecture that combines a policy (the actor) with a value function (the critic). The critic evaluates the actor's actions, providing a lower-variance signal for policy updates.

• Actor: The parameterized policy π(a|s; θ) that selects actions.
• Critic: A value function V(s; w) or Q(s,a; w) that estimates the expected return, used as a baseline.
• Advantage Function: Often uses the advantage A(s,a) = Q(s,a) - V(s) to indicate how much better an action is than average. This significantly reduces variance compared to raw returns.

A state-of-the-art policy gradient algorithm designed for stability and ease of tuning. It prevents destructively large policy updates by clipping the objective function.

• Clipped Surrogate Objective: Constrains the policy update by clipping the probability ratio between the new and old policy, preventing it from moving too far.
• Trust Region: Implicitly creates a trust region, ensuring the new policy does not deviate excessively from the old one, leading to more reliable training.
• Widely Adopted: Known for its robustness and performance across a wide range of continuous and discrete control benchmarks.

A precursor to PPO that more rigorously enforces a trust region constraint. It maximizes policy performance subject to a constraint on the KL-divergence between the new and old policies.

• Theoretical Guarantee: Uses conjugate gradient and Fisher information matrix to approximate a natural policy gradient, theoretically guaranteeing monotonic improvement.
• Computationally Complex: The constraint optimization and matrix computations are more complex than PPO's clipping approach.
• Foundation: Provides the theoretical motivation for the simpler heuristic used in PPO.

An actor-critic algorithm designed for continuous action spaces. It combines insights from DQN with a deterministic policy gradient.

• Deterministic Policy: The actor outputs a specific action, not a probability distribution, making it suitable for fine-grained control.
• Off-Policy: Uses a replay buffer and target networks (like DQN) for stable learning from past experience.
• Ornstein-Uhlenbeck Noise: Often employs this temporally correlated noise process during exploration to encourage effective action-space coverage.

An off-policy actor-critic algorithm that maximizes both expected return and policy entropy. This encourages exploration and leads to more robust policies.

• Entropy Regularization: Adds an entropy term to the reward, incentivizing the policy to be more stochastic (explorative) unless there is a clear benefit to being deterministic.
• Automatic Temperature Tuning: Can automatically adjust the entropy regularization coefficient to match a target entropy level.
• State-of-the-Art: Known for its sample efficiency and stability, particularly in complex continuous control tasks.

Reinforcement Learning is a machine learning paradigm where an agent learns to make sequential decisions by interacting with an environment. The agent receives rewards (or penalties) for its actions and aims to discover a policy—a mapping from states to actions—that maximizes its long-term cumulative reward. It is the foundational framework within which policy gradient methods operate.

• Core Components: Agent, Environment, State, Action, Reward, Policy.
• Key Challenge: Balancing exploration (trying new actions) with exploitation (using known good actions).
• Applications: Game playing (AlphaGo), robotic control, autonomous vehicle navigation, and dynamic resource allocation.

Actor-critic methods are a hybrid reinforcement learning architecture that combines the strengths of policy-based and value-based approaches. The system has two components:

• Actor: A parameterized policy (e.g., a neural network) that selects actions. It is updated using policy gradient methods.
• Critic: A value function (e.g., another neural network) that estimates the expected return from a given state or state-action pair. It critiques the actor's actions by providing a lower-variance estimate of the advantage.

This separation allows for more stable and efficient learning than pure policy gradient methods, as the critic reduces the high variance of reward signals. Algorithms like A3C (Asynchronous Advantage Actor-Critic) and PPO (Proximal Policy Optimization) are prominent actor-critic variants.

The REINFORCE algorithm, also known as the Monte Carlo policy gradient, is one of the simplest and most foundational policy gradient methods. It operates by:

1. Running an episode (or a batch of episodes) following the current stochastic policy.
2. Calculating the total return from each time step to the end of the episode.
3. Updating the policy parameters by ascending the gradient of the log-probability of the taken actions, weighted by the obtained return.

• Formula: ∇J(θ) ≈ Σ_t (G_t ∇_θ log π_θ(a_t | s_t)), where G_t is the return.
• Characteristic: It is a Monte Carlo method, requiring complete episodes before an update, which can lead to high variance in the gradient estimates.
• Foundation: REINFORCE is the basis from which more advanced policy gradient methods with variance reduction techniques (like actor-critic) are derived.

Proximal Policy Optimization (PPO) is a state-of-the-art policy gradient algorithm designed for stability and ease of implementation. It addresses a key challenge in policy gradient training: ensuring that policy updates are not too large, which can lead to performance collapse.

PPO's core innovation is its objective function, which uses a clipped surrogate objective. This objective penalizes changes to the policy that move the probability ratio (new policy / old policy) outside a specified interval, preventing destructively large updates.

• Key Feature: Trust Region optimization via clipping, making it more sample-efficient and robust than vanilla policy gradients.
• Advantage: It often requires less hyperparameter tuning than earlier methods like TRPO (Trust Region Policy Optimization).
• Ubiquity: PPO is a default choice for many complex RL benchmarks and real-world applications due to its reliability.

A value function is a core concept in reinforcement learning that estimates the expected long-term return (cumulative reward) from a given state or state-action pair. It is critical for evaluating the quality of states and actions, distinct from the policy which prescribes actions.

• State-Value Function V(s): Estimates the expected return starting from state s and following policy π thereafter.
• Action-Value Function Q(s, a): Estimates the expected return starting from state s, taking action a, and then following policy π.
• Role in Policy Gradients: Value functions are used as baselines to reduce the variance of policy gradient estimates. The advantage function, A(s,a) = Q(s,a) - V(s), tells the agent how much better a specific action is compared to the average action in that state, providing a more informative signal for policy updates.

A Markov Decision Process is the formal mathematical framework used to model the decision-making environment in reinforcement learning. An MDP is defined by the tuple (S, A, P, R, γ):

• S: A set of states.
• A: A set of actions.
• P: State transition probability function, P(s' | s, a).
• R: Reward function, R(s, a, s').
• γ: Discount factor (0 ≤ γ ≤ 1) for future rewards.

The Markov Property states that the future state and reward depend only on the current state and action, not on the full history. Policy gradient methods directly optimize the policy within this MDP framework. The objective is to find parameters θ that maximize the expected discounted return J(θ) = E[Σ γ^t R_t] under the policy π_θ.