Actor-Critic Method

The Actor is a parameterized policy function, typically a neural network, that maps the observed state of the environment to a probability distribution over possible actions. Its sole objective is to learn and improve the strategy for action selection.

• Role: Selects which action to take in a given state.
• Output: A probability distribution (for discrete actions) or parameters of a distribution like mean and variance (for continuous actions).
• Training Signal: Updated using policy gradient methods, guided by the advantage function estimated by the Critic to increase the probability of high-advantage actions.

The Critic is a parameterized value function, also a neural network, that estimates the expected cumulative future reward (the value) of being in a given state or state-action pair. It acts as an evaluator for the Actor's decisions.

• Role: Predicts the quality (value) of the current state or the chosen action.
• Output: A scalar value estimate, such as the state-value V(s) or action-value Q(s,a).
• Training Signal: Updated using temporal difference (TD) learning to minimize the error between its prediction and the actual observed return, providing a baseline or advantage for the Actor.

The Advantage Function A(s,a) is the central signal that couples the Actor and Critic. It measures how much better a specific action is compared to the average action in that state.

• Definition: A(s,a) = Q(s,a) - V(s). It quantifies the relative benefit of taking action 'a' in state 's'.
• Purpose: Provides a lower-variance gradient for the Actor than a pure reward signal. The Actor learns to increase the probability of actions with positive advantage and decrease the probability of actions with negative advantage.
• Estimation: Often computed using methods like Generalized Advantage Estimation (GAE), which provides a good bias-variance trade-off.

Temporal Difference (TD) Error is the primary learning signal for the Critic and is often used as an unbiased estimate of the advantage function. It represents the difference between the Critic's prediction and a more informed target.

• Calculation: For a state-value Critic, δ = r + γV(s') - V(s), where r is the immediate reward, γ is the discount factor, and s' is the next state.
• Role: This error δ is used to update the Critic's parameters. Simultaneously, it is frequently used as a simple estimate A(s,a) ≈ δ to train the Actor, especially in algorithms like Actor-Critic with Eligibility Traces (A2C).

Actor-Critic architectures can be designed for different data collection regimes, impacting sample efficiency and stability.

• On-Policy (e.g., A2C, PPO): The Actor (policy) used to collect training data is the same policy being optimized. This requires fresh data after each update, which can be less sample efficient but often more stable.
• Off-Policy (e.g., DDPG, TD3, SAC): The policy collecting data (a behavior policy) can differ from the target policy being optimized. This allows reuse of past experience stored in a replay buffer, greatly improving sample efficiency but adding complexity for stable training.

The training process follows a synchronous, iterative loop that integrates all components:

1. Interaction: The Actor selects an action a based on the current state s.
2. Execution: The action is performed in the environment, yielding a reward r and a new state s'.
3. Critic Evaluation: The Critic computes the TD error δ using V(s), r, and V(s').
4. Critic Update: The Critic's parameters are updated to minimize the TD error (squared).
5. Advantage Calculation: An advantage estimate A(s,a) is derived (e.g., A = δ).
6. Actor Update: The Actor's parameters are updated via the policy gradient, using A(s,a) to weight the gradient, increasing the log-probability of actions with positive advantage.

This loop forms the foundation for advanced algorithms like Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC).

Advantage Actor-Critic (A2C) is a synchronous, deterministic algorithm where a centralized learner updates a global policy using gradients from multiple parallel environments. Its asynchronous variant, A3C, employs multiple actor-learners that update the global model asynchronously, removing the need for an experience replay buffer. Both use the advantage function (A = Q - V) as the critic's feedback, which reduces variance in policy updates compared to using total returns.

• Core Mechanism: The critic estimates the state-value function V(s). The actor's update is proportional to the advantage A(s,a), pushing the policy toward actions better than the baseline.
• Key Benefit: Strikes a balance between sample efficiency (using a value baseline) and conceptual simplicity.

Proximal Policy Optimization (PPO) is an on-policy actor-critic algorithm designed for stable, reliable training with simple implementation. It prevents destructively large policy updates by using a clipped surrogate objective function. The core idea is to constrain the probability ratio between the new and old policies, ensuring updates stay within a trusted region.

• Clipped Objective: The primary loss includes a clip term: L = min( r(θ)A, clip(r(θ), 1-ε, 1+ε)A ), where r(θ) is the probability ratio and A is the estimated advantage.
• Common Use Case: The de facto standard RL optimizer in Reinforcement Learning from Human Feedback (RLHF) pipelines for aligning large language models due to its stability.

Soft Actor-Critic (SAC) is an off-policy, maximum entropy actor-critic algorithm designed for continuous action spaces. It augments the standard reward with an entropy term, encouraging the policy to explore more broadly while preventing premature convergence. The algorithm concurrently learns a policy (actor), a soft Q-function (critic), and a state-value function.

• Maximum Entropy RL: The objective is to maximize expected reward plus policy entropy: J(π) = Σ E[ r + α H(π(·|s)) ]. The temperature parameter α balances exploration and exploitation.
• Key Features: Off-policy training (improves sample efficiency), automatic entropy tuning, and strong performance on robotic control benchmarks.

Twin Delayed DDPG (TD3) is a robust, off-policy actor-critic algorithm that addresses function approximation errors in Deep Deterministic Policy Gradient (DDPG). It introduces three key improvements: 1) Clipped Double Q-Learning, using the minimum of two critic networks to mitigate overestimation bias; 2) Target Policy Smoothing, adding noise to the target action to reduce value function variance; and 3) Delayed Policy Updates, updating the actor less frequently than the critic.

• Architecture: Employs six networks: two critics, two target critics, an actor, and a target actor.
• Primary Application: Excels in environments with continuous, high-dimensional action spaces, such as robotic locomotion.

Trust Region Policy Optimization (TRPO) is a precursor to PPO that more rigorously enforces a trust region constraint. It guarantees monotonic policy improvement by using a second-order optimization method (conjugate gradient) to ensure the Kullback-Leibler (KL) divergence between successive policies remains below a threshold.

• Core Constraint: Maximizes the surrogate objective subject to a hard KL-divergence constraint: 𝔼[KL(π_old || π_new)] ≤ δ.
• Trade-off: Provides stronger theoretical guarantees than PPO but is computationally more complex due to the need for Fisher Information Matrix approximations and conjugate gradient steps.

Deterministic Policy Gradient (DPG) provides the theoretical foundation for learning deterministic policies in continuous action spaces. Its deep learning extension, Deep DPG (DDPG), is an off-policy actor-critic algorithm that combines a deterministic actor with a Q-function critic, using a replay buffer and target networks for stability.

• Key Formula: The policy gradient is the gradient of the Q-value with respect to the actions: ∇_θ J ≈ 𝔼[ ∇_a Q(s,a) ∇_θ π(s) ].
• Algorithm Components: Employs experience replay for decorrelating samples and soft target updates (polyak averaging) to slowly track the learned networks, preventing divergence.

Proximal Policy Optimization (PPO) is an on-policy reinforcement learning algorithm that uses an actor-critic architecture with a clipped objective function to ensure stable training. It is the dominant RL optimizer used in the RLHF pipeline to fine-tune language models based on reward model scores.

• Mechanism: Constrains policy updates by clipping the probability ratio between new and old actions, preventing destructively large steps that can collapse performance.
• Role in RLHF: The actor is the language model policy being optimized, and the critic is the value function estimating the expected future reward. PPO uses the advantage estimate (from the critic) to update the actor.

Reinforcement Learning from Human Feedback (RLHF) is the primary alignment pipeline where actor-critic methods are applied. It trains a language model to generate outputs preferred by humans.

• Pipeline Stages: 1) Supervised Fine-Tuning (SFT) creates an initial policy. 2) A Reward Model is trained on human preference data. 3) RL Optimization (e.g., PPO) uses the reward model to train the SFT model, which acts as the actor. A critic (value function) is trained simultaneously to estimate the expected reward.
• Actor-Critic's Role: The fine-tuned LLM is the actor proposing actions (text continuations). The critic evaluates the state-value of those actions to compute advantages for the PPO update.

A Reward Model is a neural network trained to predict a scalar reward for a given language model output, typically from pairwise human preference data using the Bradley-Terry model.

• Function in Actor-Critic: In RLHF, the reward model provides the primary training signal. It is not the critic. Instead, the critic (or value function) is a separate network trained to estimate the expected cumulative reward from a given state, which is used to compute the advantage for policy gradient updates.
• Key Difference: The reward model gives a per-output score. The critic estimates the long-term value of being in a given state (partial text), enabling more stable credit assignment.

The Advantage Function, A(s,a), is a core concept in actor-critic methods that measures how much better a specific action is compared to the average action in a given state.

• Calculation: (A(s,a) = Q(s,a) - V(s)). (Q(s,a)) is the action-value (expected return from taking action a in state s). (V(s)) is the state-value estimated by the critic.
• Purpose: It reduces variance in policy gradient updates. The actor is updated to increase the probability of actions with positive advantage and decrease the probability of actions with negative advantage.
• Generalized Advantage Estimation (GAE): A common technique to compute a low-variance, bias-controlled advantage estimate by exponentially weighting k-step temporal difference errors.

Direct Preference Optimization (DPO) is an alternative to the actor-critic RLHF pipeline. It directly optimizes a language model policy on preference data using a closed-form loss, bypassing the need for a separate reward model and RL loop.

• Contrast with Actor-Critic: DPO reformulates the RL problem as a supervised loss. It does not require training a critic network or performing iterative policy gradient updates (like PPO).
• Efficiency: This makes DPO simpler and more stable to train but is typically performed offline on a fixed dataset. Actor-critic methods like PPO can be used in online settings where the policy interacts with a dynamic environment or reward signal.

Trust Region Policy Optimization (TRPO) is a precursor to PPO that more rigorously enforces a trust region constraint to ensure stable policy updates in actor-critic methods.

• Core Idea: It limits the KL divergence between the new and old policy to a predefined threshold, guaranteeing monotonic improvement (theoretically).
• Comparison to PPO: PPO approximates this trust region constraint with its clipped surrogate objective, which is simpler to implement and tune. Both are actor-critic algorithms where the actor is the policy and the critic is a value function used to compute advantages.
• Use Case: While largely superseded by PPO in practice, TRPO's theoretical foundations are critical for understanding stable RL optimization.