Actor-Critic Method

The Actor is a parameterized policy function, typically a neural network, that maps the observed state of the environment to a probability distribution over possible actions. Its sole objective is to learn which actions to take to maximize cumulative reward.

• Output: A probability distribution (e.g., for discrete actions) or the parameters of a distribution (e.g., mean and variance for continuous actions).
• Training Signal: It is updated using policy gradient methods, guided by the advantage estimate provided by the Critic. This tells the Actor how much better or worse a selected action was compared to the average.
• Example: In a robotic control task, the Actor's network outputs the torque values for each joint motor.

The Critic is a parameterized value function, also a neural network, that estimates the expected cumulative reward (the value) of being in a given state or of taking a specific action in a state. It acts as a learned evaluator.

• Primary Forms: It can estimate the state-value function V(s) (value of a state) or the action-value function Q(s, a) (value of a state-action pair).
• Training Signal: It is trained via temporal difference (TD) learning to minimize the error between its prediction and the actual observed return (reward plus discounted future value).
• Role: By providing a baseline, it reduces the variance of policy gradient updates, leading to more stable and efficient learning than pure policy gradient methods.

The Advantage Function A(s, a) is the central signal that couples the Actor and Critic. It measures how much better a specific action a is compared to the average action in state s.

• Definition: A(s, a) = Q(s, a) - V(s). It represents the relative benefit of taking action a in state s.
• Calculation: Often estimated by the Generalized Advantage Estimation (GAE) algorithm, which provides a low-variance, bias-controlled advantage estimate across multiple timesteps.
• Purpose: This scalar value is the primary feedback used to update the Actor's policy parameters. A positive advantage increases the probability of that action, while a negative advantage decreases it.

Temporal Difference (TD) Error is the fundamental signal used to train the Critic. It represents the discrepancy between the Critic's prediction and a more informed target value.

• Formula (for V(s)): δ = r + γ * V(s') - V(s), where r is the immediate reward, γ is the discount factor, and s' is the next state.
• Interpretation: This error is the one-step advantage estimate. A large positive TD error indicates the state (or action) was better than expected.
• Dual Use: The TD error δ is directly used to update the Critic's value estimates. It also serves as an unbiased, albeit high-variance, estimate of the advantage A(s, a) for updating the Actor in some simpler Actor-Critic variants.

Entropy Regularization is a technique added to the Actor's objective function to encourage exploration by preventing the policy from becoming too deterministic too quickly.

• Mechanism: It adds a bonus proportional to the entropy of the policy's action distribution to the reward the Actor seeks to maximize. Higher entropy means more uniform, exploratory distributions.
• Benefit: This prevents mode collapse in the policy, where the agent prematurely commits to a sub-optimal action and stops exploring alternatives. It is a critical component in modern algorithms like Soft Actor-Critic (SAC) and PPO.
• Trade-off: Controlled by a temperature parameter α that balances the reward maximization objective with the exploration incentive.

In practical implementations, the Actor and Critic networks often share lower-level layers that function as a feature extractor, processing raw state observations (e.g., pixels from a game screen).

• Architecture: A common convolutional or dense backbone network processes the input. Its output features are then fed into separate Actor and Critic "heads"—smaller networks that produce the policy distribution and value estimate, respectively.
• Advantage: This is computationally efficient, as features are learned once and reused. It also stabilizes learning, as both networks build on a common, consistent representation of the state.
• Consideration: While efficient, it can lead to conflicting gradients if the tasks of the two heads are too dissimilar, which is sometimes mitigated by using separate networks.

The synchronous, deterministic counterpart to A3C. It waits for all parallel actors to finish a segment of experience before performing a single, averaged update to the global parameters.

• Core Mechanism: Collects trajectories from multiple environments, computes advantages, and performs a synchronized gradient update.
• Key Innovation: While simpler than A3C, it can be more efficient on GPU hardware where batch operations are optimized.
• Practical Note: Often yields comparable performance to A3C with more reproducible results due to its synchronous nature.

A family of Reinforcement Learning algorithms that optimize a parameterized policy directly by ascending the gradient of expected reward. The Actor in an Actor-Critic architecture is typically optimized using a policy gradient.

• REINFORCE is a foundational policy gradient algorithm that uses Monte Carlo returns.
• Advantage Actor-Critic (A2C/A3C) combines policy gradients with a learned value function (the Critic) to reduce variance.
• The policy gradient theorem provides the mathematical foundation for these updates.

A specific, widely adopted Actor-Critic algorithm designed for stable and sample-efficient training. It introduces a clipped surrogate objective to prevent destructively large policy updates.

• Uses a ratio between new and old policies to measure update size.
• Implements a clipping mechanism to constrain this ratio, ensuring updates are proximal.
• Often employs multiple epochs of minibatch updates from a single batch of experience, improving data efficiency. It is a default choice for many RL applications due to its robustness.

An off-policy Actor-Critic algorithm that incorporates maximum entropy reinforcement learning. The actor aims to maximize expected reward while also maximizing policy entropy, leading to improved exploration and robustness.

• Maintains separate networks for the Actor (policy), Critic (Q-function), and a value function.
• The temperature parameter automatically adjusts to balance reward and entropy objectives.
• Being off-policy, it can reuse past experience from a replay buffer, making it highly sample-efficient for continuous control tasks.

An off-policy Actor-Critic algorithm for continuous action spaces, designed as a successor to Deep Deterministic Policy Gradient (DDPG). It addresses function approximation error and overestimation bias in the critic.

• Employs twin Q-networks (two critics) and takes the minimum of their estimates to mitigate overestimation.
• Introduces delayed policy updates, where the actor is updated less frequently than the critic.
• Adds target policy smoothing, injecting noise into the target action to regularize the Q-function. It is known for its stability and high performance.

A technique used to compute a low-variance, low-bias estimate of the advantage function, which is central to many Actor-Critic algorithms. The advantage (A) measures how much better an action is than the average action in a state.

• Defined as: A(s,a) = Q(s,a) - V(s).
• GAE provides a tunable blend between Monte Carlo (high variance, zero bias) and TD(λ) estimates.
• It is a hyperparameter that controls the bias-variance trade-off. A value of λ=0 gives a 1-step TD advantage, while λ=1 gives a Monte Carlo advantage.

A policy optimization algorithm that enforces a trust region constraint to ensure monotonic improvement. It is a precursor to PPO and provides a theoretical foundation for stable policy updates.

• Optimizes a surrogate objective subject to a constraint on the Kullback–Leibler (KL) divergence between the new and old policies.
• Uses the conjugate gradient algorithm and Fisher information matrix to approximate the natural policy gradient, making updates computationally intensive.
• PPO was developed as a simpler, first-order approximation that achieves similar performance without the complex constraint-solving.