Actor-Critic

The core design features two distinct components. The Actor is a policy network (π(a|s)) that maps states to actions or action probabilities. The Critic is a value function network (V(s) or Q(s,a)) that estimates the expected future reward from a given state or state-action pair. This separation allows for specialized learning: the actor focuses on improving action selection, while the critic provides a learned, low-variance training signal.

Actor-Critic methods are a subclass of Policy Gradient algorithms. The actor's parameters (θ) are updated by ascending the gradient of expected reward. The key formula is ∇J(θ) ≈ E[∇ log π(a|s; θ) * A(s,a)], where A(s,a) is the advantage function. The critic's primary role is to estimate this advantage, which measures how much better a specific action is compared to the average action in that state, reducing the variance of the policy gradient.

The critic is typically trained using Temporal Difference (TD) methods, such as TD(0) or TD(λ). Instead of waiting for a complete episode to calculate returns, the critic learns from the difference between its current prediction and a bootstrapped target (e.g., r + γV(s')). This enables online and incremental learning, making Actor-Critic methods more sample-efficient than pure Monte Carlo policy gradient methods, which require full episode returns.

The critic's estimates introduce a small amount of bias because they are based on its own imperfect predictions (bootstrapping). However, this is traded for a significant reduction in the variance of the policy gradient updates compared to using Monte Carlo returns. This stabilized training is a primary advantage, allowing for more consistent learning progress, especially in environments with long time horizons or sparse rewards.

The basic Actor-Critic is on-policy, meaning the critic evaluates the current actor's policy. However, significant variants exist:

• A3C/A2C: Asynchronous/Advantage Actor-Critic, classic on-policy methods.
• Soft Actor-Critic (SAC): An off-policy variant that maximizes entropy for enhanced exploration and uses a replay buffer for data efficiency.
• TD3: Twin Delayed Deep Deterministic Policy Gradient, an off-policy algorithm for continuous action spaces that addresses overestimation bias in the critic.

In Visuomotor Control Policies, Actor-Critic architectures are pivotal for learning from high-dimensional visual inputs. The actor network often incorporates a convolutional encoder to process pixel observations. The critic evaluates the encoded state representation, providing a dense learning signal for the visuomotor policy. This framework is essential for training robots to perform complex manipulation and navigation tasks directly from camera feeds using reinforcement learning.

Advantage Actor-Critic (A2C) is a synchronous, on-policy algorithm where multiple parallel actors collect trajectories. The core innovation is using the advantage function, A(s,a) = Q(s,a) - V(s), as the critic's feedback. This measures how much better a specific action is than the average action in that state. For visuomotor policies, this helps the actor discern which subtle motor adjustments are most valuable. Updates are performed synchronously after a fixed number of steps, providing stable but potentially sample-inefficient learning.

Asynchronous Advantage Actor-Critic (A3C) is the asynchronous predecessor to A2C. Multiple actor-learners interact with independent copies of the environment in parallel and asynchronously update a global shared model. This parallelism decorrelates the data, acting as a replacement for experience replay. While largely superseded by more stable synchronous methods, A3C was foundational in demonstrating efficient, GPU-accelerated RL training. Its asynchronous nature can lead to non-stationary training data but enables rapid exploration.

Proximal Policy Optimization (PPO) is arguably the most widely adopted Actor-Critic algorithm in robotics and visuomotor control. It introduces a clipped surrogate objective to prevent destructively large policy updates. The critic estimates the value function, and the actor's update is constrained to a trust region. Key features for physical systems include:

• Stability: The clipping mechanism prevents catastrophic performance drops.
• Efficiency: It performs multiple epochs of minibatch updates on a set of trajectories.
• Robustness: Works well with high-dimensional visual inputs and continuous action spaces common in control.

Soft Actor-Critic (SAC) is an off-policy, maximum entropy Actor-Critic algorithm designed for continuous control. It maximizes both expected reward and policy entropy, encouraging exploration. Key components are:

• An Actor that outputs a stochastic policy (e.g., a Gaussian distribution).
• Two Q-function (Critic) networks to mitigate overestimation bias.
• A learnable temperature parameter to automate entropy tuning. SAC's off-policy nature, using a replay buffer, makes it highly sample-efficient—a critical advantage for real-world robot learning where data collection is expensive. It excels in dexterous manipulation tasks requiring sustained exploration.

Twin Delayed Deep Deterministic Policy Gradient (TD3) is a deep deterministic policy gradient (DDPG) variant that addresses its overestimation bias and instability. Though often categorized as an Actor-Critic, it uses a deterministic actor. Its core innovations are:

• Twin Q-networks: Two critic networks trained independently; the smaller Q-value is used for the policy update to reduce overestimation.
• Target Policy Smoothing: Adds noise to the target action to smooth out Q-values.
• Delayed Policy Updates: The actor is updated less frequently than the critics. This results in more stable and reliable Q-learning, making TD3 a strong baseline for visuomotor tasks with continuous, high-precision action spaces.

Trust Region Policy Optimization (TRPO) is the direct precursor to PPO. It rigorously enforces a trust region constraint using the Kullback–Leibler (KL) divergence between the old and new policies to ensure monotonic improvement. The update is solved via a constrained optimization problem (using conjugate gradient and Fisher Information Matrix approximations). While theoretically grounded and powerful for stable training—especially in sensitive physical systems—TRPO is computationally complex. PPO was developed as a simpler, more heuristic approximation that often matches or exceeds TRPO's performance with lower computational overhead.

A class of reinforcement learning algorithms that optimize a policy by directly estimating the gradient of expected reward with respect to the policy parameters. Unlike value-based methods, they learn a parameterized policy for action selection.

• Direct Optimization: Adjusts policy parameters to increase the probability of high-reward trajectories.
• On-Policy: Typically requires fresh samples from the current policy for each update.
• High-Variance Gradients: The core challenge is reducing the variance of gradient estimates, which Actor-Critic methods address by using a value function (the critic) as a baseline.

A specific, highly popular policy gradient algorithm that improves training stability. It is a primary example of an advanced Actor-Critic method.

• Clipped Surrogate Objective: Prevents destructively large policy updates by clipping the probability ratio between new and old policies.
• Trust Region Method: Implicitly constrains updates to stay within a region where approximations are valid.
• Empirical Success: Known for its reliability and ease of tuning, making it a default choice for complex environments including robotic control and game playing.

An off-policy Actor-Critic algorithm that maximizes both expected reward and policy entropy. It is designed for continuous action spaces and promotes robust exploration.

• Maximum Entropy RL: The policy aims to be as random as possible while still achieving high reward, leading to better exploration and improved robustness to environment variations.
• Off-Policy: Efficiently reuses past experience from a replay buffer, improving sample efficiency.
• Automatic Temperature Tuning: Dynamically adjusts the entropy term to match a target entropy level, simplifying hyperparameter tuning.

The broader reinforcement learning paradigm to which Actor-Critic belongs. Agents learn a policy or value function directly from environment interaction, without explicitly learning or using a dynamics model.

• Trial-and-Error Learning: The agent learns by taking actions and observing rewards and state transitions.
• Contrast with Model-Based: Does not attempt to predict the next state; focuses on learning the control policy directly.
• Sample Inefficiency: A key limitation, as learning complex behaviors often requires a vast number of environment interactions. Actor-Critic methods aim to improve efficiency within this paradigm.

A core component of the critic in Actor-Critic architectures. It estimates the expected cumulative reward from a given state (or state-action pair), guiding the actor's policy updates.

• State-Value Function V(s): Estimates the expected return starting from state s and following the policy.
• Action-Value Function Q(s, a): Estimates the expected return after taking action a in state s and thereafter following the policy.
• Baseline for Variance Reduction: The critic's value estimate is subtracted from the observed returns to create an advantage function, which significantly reduces the variance of policy gradient updates.

A critical training technique used with off-policy Actor-Critic algorithms like DDPG and SAC. It stores past experiences in a buffer for later sampling.

• Breaks Temporal Correlations: Random sampling from the buffer decorrelates sequential experiences, leading to more stable training.
• Improves Data Efficiency: Allows the same experience to be used for multiple learning updates.
• Replay Buffer: Stores tuples of (state, action, reward, next state, done flag). The critic is typically trained via Temporal Difference (TD) learning on mini-batches sampled from this buffer.