Actor-Critic Architecture

The architecture's defining feature is the separation of the policy function (actor) and the value function (critic) into distinct, trainable components. The actor is responsible for selecting actions, mapping states to action probabilities. The critic evaluates the quality of the actor's chosen action by estimating the state-action value (Q-value) or state value (V-value). This decoupling allows each network to specialize, leading to lower variance policy updates compared to pure policy gradient methods like REINFORCE.

A primary advantage is the use of the critic's value estimate as a baseline in the policy gradient. The actor's update rule is proportional to the advantage function, which is the difference between the observed return and the critic's predicted value. By subtracting this baseline, the variance of the gradient estimates is significantly reduced. This leads to more stable and consistent learning progress, as updates are less noisy. The critic essentially provides a learned, adaptive point of comparison for each action.

Actor-Critic methods are naturally suited for environments with continuous action spaces, such as robotic control. The actor network can parameterize a probability distribution (e.g., a Gaussian) over continuous actions, outputting a mean and standard deviation. This allows for smooth, fine-grained control. In contrast, value-based methods like DQN require discretization of the action space, which becomes intractable for high-dimensional continuous control. Algorithms like Deep Deterministic Policy Gradient (DDPG) and Soft Actor-Critic (SAC) are prominent Actor-Critic variants designed for continuous control.

The critic is typically trained using Temporal-Difference (TD) learning, most commonly the TD-error. The TD-error is the difference between the critic's current prediction and the bootstrapped target (immediate reward plus discounted future value estimate). This enables online learning—the agent can update after every time step without waiting for an episode to finish. This is more sample-efficient than Monte Carlo methods. The TD-error serves a dual purpose: it is the loss signal for training the critic and forms the core of the advantage estimate used to train the actor.

The framework supports both major learning paradigms. On-policy variants, like Advantage Actor-Critic (A2C) and Proximal Policy Optimization (PPO), learn from experience collected by the current policy. They are generally more stable but less sample-efficient. Off-policy variants, like Deep Deterministic Policy Gradient (DDPG) and Soft Actor-Critic (SAC), can learn from experience stored in a replay buffer, including data from older policies. This improves sample efficiency by reusing past data but introduces additional complexity to ensure stability, often requiring target networks.

Modern Actor-Critic implementations are rarely standalone. They are enhanced by several key techniques to achieve state-of-the-art performance:

• Experience Replay: Storing and randomly sampling past transitions to break temporal correlations and improve data efficiency.
• Target Networks: Using slowly updated copies of the actor and critic networks to generate stable training targets, mitigating divergence.
• Entropy Regularization: Adding a bonus for policy entropy to the reward, encouraging exploration and preventing premature convergence to suboptimal policies (central to SAC).
• Parallelized Rollouts: As seen in A2C, using multiple environment instances to collect diverse, decorrelated data batches in parallel.

A3C is a foundational parallelized algorithm where multiple agent instances (workers) interact with independent copies of the environment asynchronously. Each worker computes gradients for its local copy of the actor and critic networks, which are then used to asynchronously update a global network. This parallelization decorrelates the data, enabling stable training without an experience replay buffer. Key features include:

• Asynchronous updates eliminate the need for experience replay.
• Uses the Advantage function (A = Q(s,a) - V(s)) to reduce variance in policy updates.
• Inherently explores different parts of the state space via parallel workers.

A2C is the synchronous, deterministic counterpart to A3C. Instead of asynchronous updates, all parallel workers synchronize their gradients before applying a single update to the global network. This often leads to more coherent updates and can be more efficient on hardware that benefits from synchronized operations. Key features include:

• Synchronous updates provide a more stable gradient estimate.
• Simpler implementation and often more reproducible than A3C.
• Still utilizes the Advantage function for lower-variance policy gradients.

SAC is an off-policy, maximum entropy algorithm that is a de facto standard for continuous control tasks. It modifies the standard Actor-Critic objective to maximize both expected reward and the entropy of the policy, encouraging robust exploration. Key features include:

• Maximum Entropy RL: The policy aims to be as random as possible while achieving high reward.
• Off-Policy Learning: Uses a replay buffer for sample efficiency.
• Employs a learnable temperature parameter to automatically tune the entropy term.
• Typically uses a double Q-network critic to prevent overestimation bias.

TD3 is an off-policy Actor-Critic algorithm designed as a direct successor to DDPG, specifically addressing its tendency to overestimate Q-values. It introduces three key improvements for stable learning in continuous action spaces. Key features include:

• Twin (Clipped Double) Q-Learning: Uses two critic networks and takes the minimum of their estimates to mitigate overestimation.
• Target Policy Smoothing: Adds noise to the target action to smooth out Q-value estimates.
• Delayed Policy Updates: The actor (policy) network is updated less frequently than the critics.

While often classified as policy gradient methods, TRPO and PPO can be viewed as Actor-Critic algorithms where the value function (critic) is used to compute advantages. They focus on making stable, monotonic policy improvements by constraining the size of policy updates. Key distinctions:

• TRPO uses a complex second-order optimization with a KL-divergence constraint to ensure updates stay within a trust region.
• PPO simplifies this with a clipped surrogate objective that penalizes large changes, making it easier to implement and tune. PPO often uses a value function critic to compute advantages for the policy (actor) update.

DDPG is an off-policy Actor-Critic algorithm for continuous action spaces. It combines insights from DQN with a deterministic policy gradient. The actor outputs a deterministic action, and the critic learns the Q-value of that state-action pair. Core mechanisms include:

• Replay Buffer: Stores experience for off-policy learning.
• Target Networks: Uses slowly updated copies of the actor and critic networks (soft updates) to stabilize training.
• Deterministic Policy Gradient Theorem: Provides the theoretical foundation for updating the actor using the gradient of the critic.

A foundational class of algorithms upon which the Actor is built. Policy Gradient Methods directly optimize a parameterized policy function using gradient ascent on the expected cumulative reward. Unlike value-based methods, they learn a probability distribution over actions.

• Direct Policy Optimization: The policy's parameters are updated to increase the probability of actions that lead to higher returns.
• High-Dimensional/Action Spaces: Particularly effective for continuous action spaces, common in robotics control.
• Vanilla Policy Gradient: The basic REINFORCE algorithm is a Monte Carlo policy gradient method, which can have high variance. Actor-Critic architectures reduce this variance by using the Critic as a baseline.

A dominant, on-policy Actor-Critic algorithm known for its stability and ease of use. PPO improves training stability by limiting how much the policy can change in a single update.

• Clipped Surrogate Objective: The core innovation. It constrains the policy update by clipping the probability ratio, preventing destructively large steps.
• Actor-Critic Core: Uses separate networks for the policy (Actor) and value function (Critic). The Critic estimates the value of states to reduce variance in the Actor's updates.
• Robust Default: Its reliability has made PPO a go-to algorithm for a wide range of continuous control benchmarks in environments like MuJoCo and PyBullet.

An off-policy, maximum entropy Actor-Critic algorithm designed for sample efficiency and robust exploration in continuous action spaces. SAC aims to maximize both expected reward and policy entropy.

• Maximum Entropy RL: Encourages the policy to be more stochastic, leading to better exploration and improved robustness to hyperparameters.
• Off-Policy Learning: Uses a replay buffer, making it more sample-efficient than on-policy methods like PPO.
• Twin Q-Networks: Employs two Q-function (Critic) networks and takes the minimum of their outputs to mitigate overestimation bias, a common issue in Q-learning.
• Automatic Temperature Tuning: Dynamically adjusts the entropy term to maintain a target level of exploration.

An early and influential off-policy Actor-Critic algorithm for continuous control. DDPG combines insights from DQN with a deterministic policy gradient.

• Deterministic Policy: The Actor outputs a specific action, not a distribution. This is suitable for many physical control tasks.
• Experience Replay & Target Networks: Borrows these stabilization techniques from DQN. Uses separate, slowly-updating target networks for both Actor and Critic to reduce training instability.
• Action-Value Critic: The Critic (Q-network) evaluates the quality of the state-action pair provided by the Actor.
• Challenges: Can be sensitive to hyperparameters and prone to overfitting the Q-function.

The fundamental learning mechanism for the Critic. TD Learning is a class of model-free methods that update value estimates based on the difference between successive predictions.

• Bootstrapping: Updates estimates based on other estimates (e.g., the value of the next state). This blends ideas from Monte Carlo methods and dynamic programming.
• TD Error: The core signal. It is the difference between the current value estimate and a more informed target (e.g., reward + discounted next value). δ = r + γV(s') - V(s)
• Critic's Role: In Actor-Critic, the Critic's primary job is to accurately estimate the value function, and its learning is driven by minimizing the TD error. This estimated value is then used to critique and guide the Actor's updates.