Actor-Critic

The actor is a parameterized policy function, typically a neural network, that maps the observed state of the environment to a probability distribution over possible actions. Its sole purpose is to select which action to take. It is updated using policy gradient methods, where the gradient of expected reward is estimated and used to adjust the network's weights to increase the probability of high-value actions.

• Function: π(a | s; θ) - Selects action a given state s using parameters θ.
• Goal: Directly maximize the cumulative reward.
• Update Direction: Guided by the critic's evaluation.

The critic is a parameterized value function, also a neural network, that estimates the expected cumulative reward (the value) of being in a given state or of taking a specific action in a state. It does not select actions. Instead, it evaluates the quality of the actor's decisions. Common forms are the state-value function V(s) or the action-value function Q(s, a).

• Function: V(s; w) or Q(s, a; w) - Estimates future rewards using parameters w.
• Goal: Accurately predict the return (sum of discounted future rewards).
• Output: Provides a scalar feedback signal (the TD error) to the actor.

The core learning signal in most actor-critic methods is the Temporal Difference (TD) error. This is calculated by the critic after the actor takes an action and the environment provides a reward. The TD error represents the difference between the critic's prediction and the actual observed outcome.

• Formula (simplified): δ = r + γV(s') - V(s)
  • r: Immediate reward.
  • γ: Discount factor.
  • V(s'): Critic's value estimate for the new state.
  • V(s): Critic's value estimate for the old state.
• Role: A positive δ indicates the action was better than expected, so the actor should increase its probability. A negative δ suggests the action was worse. This δ directly scales the policy gradient used to update the actor.

A defining feature is the decoupled, simultaneous optimization of two objectives. The networks are trained in parallel within the same interaction loop with the environment.

1. Actor Update: The actor's parameters (θ) are adjusted to maximize the expected reward, using the TD error from the critic as an advantage estimate. Algorithms like the REINFORCE with baseline or more advanced ones like A2C/A3C and PPO define this update rule.
2. Critic Update: The critic's parameters (w) are adjusted to minimize the error in its value predictions, typically using a mean-squared error loss against the TD target (r + γV(s')). This is a standard regression problem.

This separation often leads to more stable and sample-efficient learning compared to pure policy gradient methods.

Many advanced actor-critic methods do not use the raw TD error directly. Instead, they estimate the advantage function A(s, a). The advantage measures how much better a specific action a is compared to the average action in state s.

• Definition: A(s, a) = Q(s, a) - V(s)
• Interpretation:
  • A(s, a) > 0: Action a is better than average.
  • A(s, a) < 0: Action a is worse than average.
• Benefit: Using the advantage reduces variance in the policy gradient update without introducing bias, leading to faster and more stable convergence. Methods like Advantage Actor-Critic (A2C) explicitly calculate this.

Actor-critic architectures can be implemented in both on-policy and off-policy learning paradigms, which dictates what data is used for updates.

• On-Policy (e.g., A2C, A3C, PPO): The critic evaluates and the actor improves the same policy that is being used to interact with the environment. Data is collected, used for an update, and then discarded. This ensures stability but can be less sample-efficient.
• Off-Policy (e.g., DDPG, TD3, SAC): The actor learns a target policy using data generated by a different behavior policy (often an exploratory version of the actor). This allows reuse of past experiences stored in a replay buffer, dramatically improving sample efficiency. The critic in these methods often learns a Q-function.

A class of reinforcement learning algorithms that directly optimize an agent's policy by ascending the gradient of expected reward with respect to the policy parameters. The actor in an actor-critic system is typically trained using a policy gradient method.

• Direct Optimization: Adjusts policy parameters to increase the probability of high-reward actions.
• Variance Reduction: The critic's value estimate is often used as a baseline to reduce the high variance of gradient estimates, a technique central to algorithms like REINFORCE with baseline.
• Examples: Includes REINFORCE, PPO (Proximal Policy Optimization), and TRPO (Trust Region Policy Optimization).

A core concept in model-free reinforcement learning where an agent learns by bootstrapping—updating its value estimates based on the difference between successive predictions. The critic in an actor-critic architecture is fundamentally a TD learner.

• Bootstrapping: Updates the value of a state using the estimated value of the next state (e.g., V(s) ← V(s) + α[r + γV(s') - V(s)]).
• Efficiency: Enables learning after every step without waiting for a complete episode's outcome, unlike Monte Carlo methods.
• Foundation: Algorithms like Q-learning and SARSA are built on TD learning principles.

A specific, highly popular policy gradient algorithm that functions as an advanced actor-critic method. It introduces constraints to prevent destructively large policy updates during training.

• Clipped Surrogate Objective: The primary innovation; it limits the change in the policy to a trusted region by clipping the probability ratio, ensuring stable updates.
• Actor-Critic Core: Uses a policy network (actor) and a value network (critic). The critic estimates the value function to compute advantages for the actor's update.
• Practical Dominance: Known for its sample efficiency, stability, and ease of implementation, making it a default choice for many continuous control tasks.

An off-policy, maximum entropy reinforcement learning algorithm designed for continuous action spaces. It modifies the standard actor-critic framework to maximize both expected reward and policy entropy.

• Entropy Maximization: Encourages exploration by rewarding the actor for behaving randomly, leading to more robust policies that capture multiple modes of optimal behavior.
• Off-Policy Learning: Uses a replay buffer of past experiences, improving sample efficiency.
• Architecture: Employs an actor (policy) network, a state-value (critic) network, and two Q-function networks to mitigate overestimation bias.

A central mathematical object in actor-critic methods, defined as A(s,a) = Q(s,a) - V(s). It measures how much better a specific action a is compared to the average action in state s.

• Credit Assignment: Provides a refined signal for the actor, indicating not just if an action was good, but how much better it was than the baseline (the state value V(s)).
• Variance Reduction: Using the advantage (A2C - Advantage Actor-Critic) instead of raw returns drastically reduces the variance of policy gradient updates, accelerating convergence.
• Calculation: Estimated by the critic using Temporal Difference error (e.g., δ = r + γV(s') - V(s)).

A fundamental dichotomy in RL that defines the relationship between the policy being evaluated/improved and the policy used to collect data. Actor-critic architectures can be designed for either paradigm.

• On-Policy (e.g., A2C, PPO): The actor (behavior policy) and the policy being optimized are the same. Data is collected, used for an update, and then discarded.
• Off-Policy (e.g., SAC, DDPG): The agent can learn a target policy using data generated by a different, older behavior policy. Enables experience replay for greater data efficiency.
• Critic's Role: In off-policy actor-critic, the critic often learns a Q-function to evaluate the quality of actions under the target policy, even when data comes from a different policy.