Actor-Critic Methods

The actor is a parameterized policy function, typically a neural network, that maps environment states to probability distributions over possible actions. Its sole objective is to learn which actions yield the highest long-term reward by directly adjusting its parameters to increase the probability of selecting advantageous actions. The actor's update is guided by the advantage function estimated by the critic, which tells the actor how much better a specific action was compared to the average action in that state. This direct policy parameterization allows the actor to learn stochastic policies, essential for exploration in environments with partial observability.

The critic is a value function estimator, also a neural network, that evaluates the quality of a given state or state-action pair. Its primary role is to learn to predict the expected cumulative future reward (the value). The critic does not select actions; instead, it provides a training signal to the actor. By estimating the state-value function V(s) or the action-value function Q(s,a), the critic reduces the variance of policy updates. It acts as a learned baseline, allowing the actor to distinguish between positive outcomes caused by its actions versus those attributable to a generally good state.

The advantage function A(s,a) is the critical signal that couples the actor and critic. It is defined as the difference between the action-value Q(s,a) and the state-value V(s): A(s,a) = Q(s,a) - V(s). This function quantifies how much better a specific action a is than the average action the policy would take in state s. The actor uses this advantage signal for its gradient updates:

• Positive Advantage: The selected action was better than average; increase its probability.
• Negative Advantage: The action was worse than average; decrease its probability. This mechanism provides a low-variance, biased estimate that dramatically stabilizes training compared to using raw returns.

Temporal Difference (TD) Error is the fundamental learning signal used to update the critic. For a transition (s, a, r, s'), the TD error is calculated as: δ = r + γV(s') - V(s), where γ is the discount factor. This error represents the difference between the critic's current prediction V(s) and the better, bootstrapped target r + γV(s'). Crucially, in many actor-critic algorithms like Advantage Actor-Critic (A2C), this TD error serves as an unbiased estimate of the advantage function A(s,a). The critic's parameters are updated to minimize this TD error, while the actor uses the same δ to adjust its policy.

The actor's update is derived from the Policy Gradient Theorem. The objective is to maximize the expected return J(θ) by ascending the gradient with respect to the policy parameters θ. The general update rule for the actor is a gradient ascent step: θ ← θ + α * ∇_θ log π_θ(a|s) * A(s,a), where α is the learning rate. The term ∇_θ log π_θ(a|s) is the score function, which indicates the direction in parameter space that increases the log-probability of the taken action. This update is scaled by the advantage A(s,a), ensuring actions with higher-than-expected reward are reinforced. This is the core mechanism of algorithms like REINFORCE with baseline, which is a simple actor-critic.

Actor-critic algorithms are implemented in two primary parallelization paradigms:

• Synchronous (A2C): Multiple actor-learners (e.g., 16) operate in parallel environments. After all actors finish a set number of steps, their gradients are averaged and a single, synchronized update is applied to a central actor and critic network. This is simpler and more stable but can be slower if one actor is delayed.
• Asynchronous (A3C): Each actor-learner thread operates completely independently with its own copy of the environment and a local copy of the model parameters. Threads periodically push their gradient updates to a global shared parameter server and pull the latest parameters. This maximizes hardware utilization and often leads to faster wall-clock time convergence, though the training dynamics are noisier.

Proximal Policy Optimization (PPO) is a policy gradient algorithm that directly optimizes the actor's policy. It uses a clipped objective function to prevent destructively large updates, ensuring stable and sample-efficient learning. PPO is the dominant algorithm used to train the actor in modern actor-critic implementations, especially when fine-tuning large language models with a learned reward signal from a critic.

• Core Mechanism: Constrains policy updates by clipping the probability ratio between new and old policies.
• Primary Use: The workhorse algorithm for policy-based RL, including in RLHF/RLAIF pipelines.
• Relation to Actor-Critic: PPO provides the optimization engine for the actor, while the critic (value function) provides the advantage estimates used in the PPO objective.

Trust Region Policy Optimization (TRPO) is a precursor to PPO that more rigorously enforces a constraint on policy updates. It uses a second-order optimization method to ensure the new policy stays within a trust region of the old policy, guaranteeing monotonic improvement. While more mathematically grounded, TRPO is computationally more complex than PPO.

• Core Mechanism: Enforces a hard constraint on the KL divergence between successive policies.
• Primary Use: Provides a stable foundation for policy gradient methods; often used in robotics and control.
• Relation to Actor-Critic: Serves as the theoretical basis for stable actor updates. Modern actor-critic methods often use PPO's simpler approximation of TRPO's trust region.

The Advantage Function, denoted as A(s, a), is a central concept that quantifies how much better a specific action is compared to the average action in a given state. It is calculated as the difference between the action-value function Q(s, a) and the state-value function V(s): A(s,a) = Q(s,a) - V(s).

• Core Mechanism: Provides a baseline that reduces the variance of policy gradient estimates.
• Primary Use: The key signal used by the actor to determine which actions to reinforce or discourage.
• Relation to Actor-Critic: The critic's primary role is to estimate V(s) or A(s,a) accurately. In the Advantage Actor-Critic (A2C) algorithm, the critic directly learns to output the advantage.

Model-Based Reinforcement Learning (MBRL) involves an agent learning an explicit internal model of the environment's dynamics (transition function) and reward function. The agent can then use this model for planning, simulating trajectories without direct environmental interaction.

• Core Mechanism: Learns a function f(s, a) -> (s', r) to predict next states and rewards.
• Primary Use: Dramatically improves sample efficiency by reducing the need for real environment steps.
• Relation to Actor-Critic: Actor-critic is typically model-free. However, hybrid architectures exist where a world model acts as a simulated environment for the actor-critic to practice in, or where the model's predictions are used to improve the critic's value estimates.

Asynchronous Advantage Actor-Critic (A3C) is a seminal distributed variant of the actor-critic architecture. Multiple worker agents interact with parallel instances of the environment, each with its own actor and critic. They asynchronously update a global shared network, enabling efficient, parallel exploration and stable learning without experience replay.

• Core Mechanism: Uses asynchronous parallel actors to decorrelate gradients and stabilize training.
• Primary Use: Enabled faster and more robust deep RL training on CPU clusters before GPU-optimized synchronous methods became dominant.
• Relation to Actor-Critic: A3C is a specific, influential implementation of the general actor-critic paradigm, highlighting how the architecture scales to parallel computation.

Soft Actor-Critic (SAC) is a state-of-the-art off-policy actor-critic algorithm designed for continuous action spaces. It incorporates entropy maximization into its objective, encouraging the policy to explore more broadly and leading to more robust learning. SAC uses a stochastic actor and typically employs two Q-function (critic) networks and their minimum to reduce overestimation bias.

• Core Mechanism: Maximizes expected reward plus policy entropy (maximum entropy RL).
• Primary Use: The go-to algorithm for complex continuous control tasks in robotics and simulation.
• Relation to Actor-Critic: Represents the modern evolution of the actor-critic framework, emphasizing off-policy sample efficiency and stochastic policies for exploration, in contrast to many on-policy, deterministic variants.