Actor-Critic

The actor is a policy network (π(a|s; θ)) that maps the current state (s) to a probability distribution over possible actions (a). Its parameters (θ) are optimized to maximize the expected cumulative reward. The actor's role is pure action selection.

• Input: State representation (e.g., sensor data, game screen).
• Output: Action probabilities (for discrete actions) or parameters of a continuous distribution (e.g., mean and variance for a Gaussian).
• Update Mechanism: Uses the policy gradient, scaled by the critic's evaluation, to adjust its parameters.

The critic is a value network (V(s; w) or Q(s,a; w)) that estimates the expected future return from a given state (or state-action pair). Its parameters (w) are learned to minimize a temporal difference (TD) error. The critic's role is pure evaluation.

• Input: State (for V(s)) or state-action pair (for Q(s,a)).
• Output: A scalar value representing the predicted quality of the state/action.
• Core Function: Provides a low-variance, baseline signal to the actor, reducing the noise inherent in Monte Carlo returns.

The advantage function is a central concept, calculated as A(s,a) = Q(s,a) - V(s). It measures how much better a specific action (a) is compared to the average action at that state. It is the primary signal used to update the actor.

• Interpretation: A(s,a) > 0 means the action is better than average; A(s,a) < 0 means it is worse.
• Role in Updates: The actor's policy gradient is weighted by the advantage. Actions with positive advantage are made more likely.
• Estimation: Often approximated directly by networks (e.g., in Advantage Actor-Critic (A2C)) or derived from TD error.

Temporal Difference (TD) Error (δ) is the fundamental learning signal for the critic and a key component for the actor. For a value critic V(s), it is defined as: δ = r + γV(s') - V(s), where r is the immediate reward, γ is the discount factor, and s' is the next state.

• Critic Update: The critic's parameters are updated to minimize the squared TD error (δ²).
• Actor Signal: In many implementations (e.g., one-step actor-critic), the TD error δ serves as an unbiased estimate of the advantage A(s,a).

The actor is updated using the policy gradient theorem. The gradient of the expected return with respect to the actor's parameters (θ) is proportional to the expected value of the gradient of the log-policy multiplied by the advantage.

• Update Rule: Δθ ∝ ∇_θ log π(a|s; θ) * A(s,a)
• Role of Critic: The critic supplies A(s,a), replacing high-variance Monte Carlo returns with a learned baseline.
• Algorithm Example: Proximal Policy Optimization (PPO) is a modern, stable actor-critic algorithm that uses a clipped objective to control the size of policy updates.

Entropy regularization is a common technique added to the actor's objective function to encourage exploration. It penalizes the policy for becoming too deterministic too early.

• Formula: The policy's objective becomes: maximize [ expected advantage + β * H(π(·|s)) ], where H is the entropy and β is a hyperparameter.
• Effect: Increases the randomness of the action distribution, preventing the actor from converging to a suboptimal policy prematurely.
• Trade-off: Controlled by β; too high encourages random exploration, too low leads to premature exploitation.

Advantage Actor-Critic (A2C) is the synchronous, deterministic variant of the A3C algorithm. It uses the advantage function—the difference between the state-action value Q(s,a) and the state value V(s)—as the critic's feedback signal.

• Core Mechanism: The advantage A(s,a) = Q(s,a) - V(s) tells the actor whether an action is better or worse than the average action in that state.
• Reduced Variance: By subtracting the baseline V(s), the policy gradient updates have lower variance than using raw returns, leading to more stable training.
• Implementation: Typically uses multiple parallel environments but performs a synchronous update after all workers finish a segment, averaging gradients. This is more efficient for GPU utilization than the asynchronous A3C.

Asynchronous Advantage Actor-Critic (A3C) is a seminal distributed algorithm where multiple worker agents interact with independent copies of the environment in parallel.

• Asynchronous Updates: Each worker computes gradients based on its own experience and asynchronously pushes updates to a global shared parameter server. This eliminates the need for an experience replay buffer.
• Speed & Diversity: Parallel exploration decorrelates the data, making it more i.i.d.-like, which helps convergence. It was groundbreaking for efficiently using multi-core CPUs.
• N-Step Returns: Workers typically use n-step returns (e.g., 20 steps) to estimate the advantage, blending Monte Carlo and Temporal Difference methods. While largely superseded by more synchronous methods for GPU efficiency, A3C established key distributed RL patterns.

Soft Actor-Critic (SAC) is an off-policy actor-critic algorithm for continuous action spaces that maximizes both expected return and entropy.

• Maximum Entropy RL: The policy is trained to maximize a trade-off between expected return and the entropy of the policy. This encourages greater exploration and results in more robust policies that can capture multiple modes of optimal behavior.
• Off-Policy Learning: Uses a replay buffer for sample efficiency, learning from past experiences.
• Twinned Q-Networks: Employs two separate Q-function (critic) networks and takes the minimum of their predictions to mitigate overestimation bias—a technique borrowed from Twin Delayed DDPG (TD3). SAC is a state-of-the-art model-free algorithm for continuous control benchmarks like MuJoCo.

Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are policy optimization algorithms that can be viewed as advanced actor-critic methods with a strong focus on update stability.

• Trust Region: TRPO theoretically constrains policy updates using the KL divergence between the old and new policy, ensuring monotonic improvement. It's computationally complex, requiring conjugate gradient descent.
• PPO's Simplicity: Proximal Policy Optimization (PPO) provides a simpler, first-order approximation to TRPO's objective. Its primary mechanism is a clipped surrogate objective that prevents the new policy from moving too far from the old policy.
• Actor-Critic Core: Both use a value function (critic) to reduce variance. PPO, in particular, is the dominant RLHF optimizer for aligning large language models due to its reliability and ease of implementation.

Deep Deterministic Policy Gradient (DDPG) is an off-policy actor-critic algorithm designed for continuous action spaces.

• Deterministic Policy: The actor network outputs a single, precise action value for a given state (a deterministic policy), unlike stochastic policies that output a distribution.
• Critic as Q-Network: The critic is a deep Q-network that estimates Q(s,a). The actor is updated by performing gradient ascent on the critic's estimated Q-value with respect to the actor parameters.
• Key Techniques: Employs target networks for both actor and critic to stabilize training, and a replay buffer for off-policy learning. It is conceptually an extension of DQN to continuous domains, combined with a deterministic policy gradient theorem.

Twin Delayed DDPG (TD3) is a direct successor to DDPG that addresses its frequent issue of overestimation bias in the Q-function.

• Twin Critics: Learns two Q-function networks independently and uses the minimum of the two values for the target in the Bellman update. This clipped double-Q learning reduces overestimation.
• Target Policy Smoothing: Adds noise to the target action when computing the target Q-value. This regularizes the critic by making it harder to exploit errors in the Q-function around similar actions.
• Delayed Updates: Updates the policy (actor) less frequently than the value function (critic). This allows the critic to become more accurate before it is used to update the actor, preventing divergence. TD3 is a robust, high-performing baseline for continuous control.

A training pipeline that aligns a model, often an LLM, with human preferences. It typically uses an actor-critic loop where:

• The actor is the language model policy generating text.
• The critic is a learned reward model trained on human preference data.
• An RL optimizer like PPO updates the actor policy using rewards from the critic, often with a KL divergence penalty to prevent excessive drift from the original model.

The dominant policy gradient algorithm used to train the actor in modern actor-critic systems, including RLHF. Its key mechanism is a clipped objective function that ensures stable updates by preventing the new policy from deviating too far from the old policy. This stability is critical when the critic (value/reward model) is also being learned simultaneously.

The process of training the critic component in preference-based actor-critic systems. A reward model is a neural network trained to predict a scalar reward, typically using datasets of pairwise comparisons. Common statistical models underpinning this include the Bradley-Terry model for pairwise data and the Plackett-Luce model for rankings. The accuracy of this model directly impacts the quality of the policy updates.

The broader subfield of RL where an agent learns from qualitative preference feedback between trajectories, instead of a predefined numeric reward signal. Actor-critic is a natural architecture for PbRL, where the critic learns to estimate the value of states or actions based on the preference feedback, and the actor uses this signal to improve. This contrasts with Inverse Reinforcement Learning (IRL), which infers a full reward function from demonstrations.

A crucial regularization term added to the reward function in actor-critic fine-tuning (e.g., RLHF). It penalizes the actor policy for diverging from a reference model (usually the initial pre-trained model). This prevents reward hacking and reward overoptimization by overly specializing to the imperfect reward model, and helps mitigate catastrophic forgetting of the model's original knowledge and capabilities.

Critical failure modes in actor-critic systems with learned reward models. Reward hacking occurs when the actor finds loopholes in the critic's reward function. Reward overoptimization happens when the actor's performance on the proxy reward improves while true performance degrades. These highlight the principal-agent problem in learned reward functions and the need for robust scalable oversight techniques like debate or iterated amplification.