Actor-Critic

The Actor is a parameterized policy function, typically a neural network, that maps the current state of the environment to a probability distribution over possible actions. Its sole objective is to learn which actions yield the highest long-term reward.

• Role: Selects actions (e.g., a robot joint torque).
• Output: A probability distribution (for discrete actions) or parameters of a distribution like mean and variance (for continuous actions).
• Learning Signal: It is updated using policy gradients, specifically the Advantage signal provided by the Critic, which indicates how much better an action was than expected.
• Example Architectures: A simple Multi-Layer Perceptron (MLP) for low-dimensional states, or a convolutional network for pixel-based inputs.

The Critic is a parameterized value function, also typically a neural network, that estimates the expected cumulative reward (the value) of being in a given state or of taking a specific action in a state.

• Role: Evaluates the quality of the Actor's decisions.
• Common Forms: State-Value function V(s) (value of a state) or Action-Value function Q(s, a) (value of a state-action pair).
• Learning Signal: It is trained via temporal-difference (TD) error, minimizing the difference between its predicted value and the actual observed reward plus the discounted value of the next state. This is a supervised regression problem.
• Output: A single scalar value representing expected future return.

The Advantage Function A(s, a) is the critical signal that couples the Actor and Critic. It measures the relative benefit of taking a specific action a in state s compared to the average action in that state.

• Definition: A(s, a) = Q(s, a) - V(s).
• Interpretation: A positive advantage means the action was better than average; a negative advantage means it was worse.
• Role in Training: The Actor's policy gradient is weighted by the Advantage. Actions with high advantage are reinforced, while those with low advantage are discouraged. This reduces variance in policy updates compared to using raw returns.
• Estimation: Often estimated directly (e.g., Generalized Advantage Estimation - GAE) using the TD error from the Critic.

Temporal-Difference Error is the primary learning signal for the Critic and is fundamental to calculating the Advantage. It represents the discrepancy between the Critic's prediction and a more informed target.

• Formula (for V(s)): δ = r + γ * V(s') - V(s) where r is the immediate reward, γ is the discount factor, and s' is the next state.
• Function: This error drives the Critic's update. Simultaneously, it serves as an unbiased, low-variance estimate of the Advantage for a single step.
• On-Policy Nature: In classic Actor-Critic, this TD target is calculated using the on-policy next state value V(s'), meaning it evaluates the current Actor's policy.

The Policy Gradient Theorem provides the mathematical foundation for updating the Actor. It states that the gradient of the expected return with respect to the policy parameters can be estimated from experience.

• Core Equation: The gradient is proportional to the expected value of the score function multiplied by the Advantage: ∇J(θ) ≈ E[∇ log π(a|s) * A(s, a)].
• Role: It allows the policy to be improved by ascending this gradient. The log probability ∇ log π(a|s) points in the direction that increases the probability of action a, scaled by how advantageous A(s, a) that action was.
• Result: This theorem enables gradient-based optimization of the policy using trajectories sampled from the environment.

Entropy Regularization is a common technique added to the Actor's objective function to encourage exploration and prevent premature convergence to a suboptimal deterministic policy.

• Concept: It adds a bonus proportional to the entropy of the policy distribution π(a|s). High entropy means high uncertainty/exploration.
• Objective: The Actor maximizes Expected Return + β * H(π), where H(π) is the entropy and β is a temperature parameter controlling exploration strength.
• Effect: The policy is incentivized to try diverse actions, especially early in training, leading to more robust discovery of optimal behaviors. It is a standard component in algorithms like Soft Actor-Critic (SAC).

A family of reinforcement learning algorithms that directly optimize a parameterized policy by ascending the gradient of expected reward. The actor in an Actor-Critic system is typically trained using a policy gradient.

• REINFORCE Algorithm: A foundational policy gradient method that uses Monte Carlo returns.
• Key Mechanism: Updates move the policy toward actions that yield higher cumulative reward.
• Trade-off: Pure policy gradients have high variance; the critic is introduced to reduce this variance by providing a baseline.

A core component in RL that estimates the expected cumulative future reward from a given state (state-value function V(s)) or from a given state-action pair (action-value function Q(s,a)). The critic in an Actor-Critic architecture is a learned value function.

• Purpose: Evaluates the quality of states or actions, guiding policy improvement.
• Temporal Difference (TD) Learning: A primary method for learning value functions from incremental backups, which Actor-Critic algorithms heavily utilize.
• Bootstrapping: The critic uses its own predictions to update its estimates, enabling more efficient, online learning.

A state-of-the-art policy gradient algorithm that is fundamentally an Actor-Critic method. It introduces a clipped objective function to ensure stable, monotonic policy improvements by preventing excessively large updates.

• Core Innovation: Uses a probability ratio between new and old policies, clipped to a small range, as part of the surrogate objective.
• Practical Impact: Known for its simplicity, robustness, and excellent performance across a wide range of continuous control benchmarks (e.g., OpenAI's MuJoCo).
• Architecture: Employs separate neural networks for the actor (policy) and critic (value function), trained concurrently.

An off-policy Actor-Critic algorithm that incorporates maximum entropy reinforcement learning. It aims to maximize expected reward while also maximizing policy entropy, leading to more robust exploration and improved stability.

• Key Feature: The actor optimizes for a trade-off between expected reward and entropy, encouraging stochastic policies.
• Mechanism: Uses a double Q-learning style critic to mitigate overestimation bias and a stochastic actor.
• Application Domain: Particularly effective for real-world robotic tasks requiring sample efficiency and stable learning from high-dimensional inputs.

A central concept in Actor-Critic learning, defined as A(s,a) = Q(s,a) - V(s). It measures how much better a specific action is compared to the average action in that state.

• Role in Actor-Critic: The critic often learns to estimate the advantage function. Policy updates for the actor are then proportional to the advantage.
• Interpretation: A positive advantage indicates the action is better than average; a negative advantage indicates it is worse.
• Generalized Advantage Estimation (GAE): A widely-used technique to compute a low-variance, biased estimate of the advantage function across multiple time steps.

An off-policy, Actor-Critic algorithm designed for continuous action spaces. It combines insights from DQN (Deep Q-Network) with a deterministic policy gradient.

• Architecture: Uses two primary networks: an actor that outputs a deterministic action and a critic that estimates the Q-value of state-action pairs.
• Key Techniques: Employs replay buffers for experience replay and target networks for stable training, borrowed from DQN.
• Use Case: Well-suited for physical control problems where actions are real-valued (e.g., torque applied to a motor).