Actor-Critic

Advantage Actor-Critic (A2C) is a synchronous, on-policy algorithm where the critic estimates the advantage function, A(s,a) = Q(s,a) - V(s). This measures how much better a specific action is compared to the average action in that state.

• Core Mechanism: The actor's policy is updated using the advantage estimate as a baseline, which reduces variance in the policy gradient compared to using raw returns.
• Synchronous: Multiple agents typically run in parallel, collecting experiences that are then averaged for a single, synchronous update to the shared networks.
• Use Case: Provides a stable foundation and is often used as a baseline for continuous control tasks and simpler, discrete environments.

Asynchronous Advantage Actor-Critic (A3C) is a decentralized, asynchronous variant of A2C designed for parallel training on CPU cores.

• Core Mechanism: Multiple worker agents, each with its own environment instance, operate asynchronously. They compute gradients independently using local network parameters and periodically push updates to a global shared network.
• Key Innovation: This asynchrony acts as a decorrelation mechanism, replacing the need for a replay buffer. Different workers are likely exploring different parts of the state space simultaneously.
• Impact: A3C demonstrated that deep RL could be trained effectively on standard multi-core CPUs, significantly increasing the accessibility of advanced policy gradient methods.

Deep Deterministic Policy Gradient (DDPG) is an off-policy actor-critic algorithm designed explicitly for continuous action spaces.

• Core Mechanism: It combines a deterministic policy gradient with techniques from DQN. The actor outputs a precise, continuous action. The critic (a Q-function) evaluates the state-action pair.
• Key Stability Features:
  • Replay Buffer: Stores experiences for off-policy learning.
  • Target Networks: Uses slowly updated copies of the actor and critic networks to provide stable training targets, mitigating divergence.
• Use Case: The go-to algorithm for tasks like robotic arm manipulation and locomotion where actions are torques or velocities.

Twin Delayed DDPG (TD3) is a direct successor to DDPG that addresses its tendency toward overestimation bias in the Q-function, which can lead to policy breakdown.

• Core Innovations:
  • Twin Critics: Maintains two separate Q-networks and uses the minimum of their estimates for the value target, reducing overestimation.
  • Target Policy Smoothing: Adds noise to the target action, making it harder for the Q-function to fit sharp, erroneous peaks (a form of regularization).
  • Delayed Policy Updates: Updates the actor (policy) less frequently than the critic, allowing the value estimate to stabilize first.
• Result: TD3 is consistently more stable and reliable than DDPG, making it a preferred choice for complex continuous control benchmarks.

Soft Actor-Critic (SAC) is an off-policy algorithm that maximizes a trade-off between expected reward and policy entropy.

• Core Principle: The maximum entropy objective encourages exploration by favoring stochastic policies that act as randomly as possible while still succeeding. Formally, it maximizes E[sum(r) + α * H(π)], where H is entropy.
• Key Features:
  • Automatically trades off exploration (high entropy) and exploitation via a learnable temperature parameter (α).
  • Uses a stochastic actor, which is often more robust than a deterministic one.
  • Employs twin critics and a replay buffer like TD3.
• Use Case: Known for its exceptional sample efficiency and robustness, SAC is a top performer in modern continuous control benchmarks and is highly relevant for sim-to-real transfer where exploration and stability are paramount.

Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are on-policy actor-critic methods focused on ensuring stable, monotonic policy improvement.

• Core Problem: Large policy updates can collapse performance. Both algorithms constrain how much the new policy can deviate from the old one.
• TRPO: Uses a complex second-order optimization (conjugate gradient) to enforce a hard Kullback–Leibler (KL) divergence constraint, guaranteeing theoretical improvement.
• PPO: Provides a simpler, first-order approximation. Its primary method uses a clipped surrogate objective that penalizes changes that would move the new policy too far from the old. This is much easier to implement and tune.
• Use Case: PPO, due to its simplicity and reliability, became one of the most widely used RL algorithms in practice, from game playing to robotics simulation training.

A foundational class of reinforcement learning algorithms that directly optimize a parameterized policy function. The actor component in an Actor-Critic system is typically trained using a policy gradient method. Key characteristics include:

• Direct Policy Optimization: Adjusts policy parameters to increase the probability of high-reward actions.
• High Variance: Pure policy gradients suffer from high variance in gradient estimates, which the critic helps reduce.
• On-Policy Learning: Often requires fresh samples from the current policy, though Actor-Critic can be adapted for off-policy use.

A core theoretical component that estimates the expected cumulative future reward. In Actor-Critic, the critic is explicitly learning a value function. There are two primary types:

• State-Value Function (V(s)): Estimates the expected return from a given state.
• Action-Value Function (Q(s, a)): Estimates the expected return from taking a specific action in a state. The critic's learned value provides a baseline or advantage estimate, which is subtracted from the observed returns to calculate a lower-variance policy gradient for the actor. This is the central mechanism that stabilizes Actor-Critic training.

A dominant, state-of-the-art on-policy Actor-Critic algorithm. PPO introduces constraints to prevent destructively large policy updates, making it remarkably stable and sample-efficient for complex tasks like robotic control. Its key innovations include:

• Clipped Surrogate Objective: Limits the policy update size by clipping the probability ratio, preventing collapse.
• Multiple Epochs of Minibatch Updates: Reuses collected data for several gradient steps, improving data efficiency.
• Generalized Advantage Estimation (GAE): A technique for estimating the advantage function with low variance, which the critic learns to predict. PPO is a direct and highly practical implementation of the Actor-Critic paradigm.

A leading off-policy Actor-Critic algorithm designed for continuous action spaces. SAC incorporates maximum entropy reinforcement learning, where the policy aims to maximize expected reward and entropy, encouraging exploration. Its architecture features:

• Separate Actor and Critic Networks: The actor outputs a stochastic policy (e.g., a Gaussian distribution).
• Twin Q-Networks (Critics): Uses two Q-function estimators to mitigate overestimation bias.
• Automatic Entropy Tuning: Dynamically adjusts the temperature parameter to control the exploration-exploitation trade-off. Because it is off-policy and uses a replay buffer, SAC is exceptionally sample-efficient, making it highly suitable for real-world robotics where data collection is costly.

An off-policy Actor-Critic algorithm for continuous control. DDPG combines insights from DQN (Deep Q-Networks) with a deterministic policy gradient. Its key components are:

• Deterministic Actor: Outputs a single, precise action value for a given state, rather than a distribution.
• Critic as Q-Network: Learns the Q-function to evaluate the actor's chosen actions.
• Target Networks and Replay Buffer: Uses slowly updated target networks for both actor and critic, and a replay buffer for off-policy learning, to stabilize training. DDPG demonstrated that the Actor-Critic framework could successfully scale to deep neural networks and complex, high-dimensional tasks, paving the way for algorithms like TD3 and SAC.

A pivotal concept that directly links the actor and critic. The advantage function, A(s, a), measures how much better a specific action is compared to the average action in that state.

• Definition: A(s, a) = Q(s, a) - V(s).
• Role in Actor-Critic: The critic learns V(s) or Q(s, a). The policy gradient used to update the actor is then proportional to the advantage estimate. A positive advantage increases the probability of action a in state s.
• Reduces Variance: By comparing against the state-value baseline V(s), the advantage function provides a much lower-variance signal for policy updates than raw returns, which is the primary source of stability in Actor-Critic methods.