Actor-Critic

The Actor is a parameterized policy function, typically a neural network, that maps the current state of the environment to a probability distribution over possible actions. Its sole objective is to learn which actions maximize long-term cumulative reward.

• Role: Directly controls the agent's behavior by selecting actions.
• Output: In discrete spaces, a probability for each action. In continuous spaces, parameters (e.g., mean and variance) of a probability distribution (like a Gaussian).
• Learning Mechanism: Updated using policy gradient methods, guided by the Advantage signal from the Critic to increase the probability of good actions.

The Critic is a parameterized value function, also typically a neural network, that estimates the expected cumulative reward (the Value) from a given state or state-action pair. It acts as an evaluator, not a controller.

• Role: Assesses the quality of the current state or the action taken by the Actor.
• Output: A scalar value representing the predicted future reward (e.g., State-Value V(s) or Action-Value Q(s,a)).
• Learning Mechanism: Updated using temporal-difference (TD) learning, minimizing the error between its prediction and the actual observed reward plus the discounted future value.

The Advantage Function A(s,a) is the central signal that couples the Actor and Critic. It quantifies how much better a specific action is compared to the average action in that state.

• Calculation: A(s,a) = Q(s,a) - V(s). It represents the relative benefit of taking action a in state s.
• Purpose: Provides a lower-variance gradient signal for the Actor. Instead of using raw total reward, the Actor learns from whether an action was better or worse than expected.
• Key Benefit: Reduces training variance, leading to more stable and faster convergence compared to pure policy gradient methods like REINFORCE.

Temporal-Difference (TD) Error is the primary learning signal for the Critic and is often used as an unbiased estimate of the Advantage function.

• Formula (for a state-value Critic): δ = r + γ * V(s') - V(s) where r is the immediate reward, γ is the discount factor, V(s') is the value of the next state, and V(s) is the value of the current state.
• Interpretation: The difference between the Critic's current prediction V(s) and the better, more informed target r + γV(s').
• Dual Use: This scalar δ is used to:
  1. Update the Critic's parameters to correct its value estimates.
  2. Serve as a proxy for A(s,a) to update the Actor's policy (i.e., if δ is positive, the action was good).

The Actor is updated using a policy gradient that is weighted by the Advantage estimate. The most common implementation is the Advantage Actor-Critic (A2C/A3C) algorithm.

• Gradient Formula: The policy parameters θ are updated via: ∇θ J(θ) ≈ E[∇θ log π(a|s; θ) * A(s,a)].
• Mechanism: This formula adjusts the policy parameters to increase the log-probability of an action proportionally to its Advantage. Actions with positive Advantage (better than average) have their probability increased.
• Synchronization: In A2C (Advantage Actor-Critic), a single agent updates global networks synchronously. In A3C (Asynchronous Advantage Actor-Critic), multiple agent workers update asynchronously, improving exploration and hardware utilization.

While not exclusive to Actor-Critic, these components are critical for stabilizing training in modern implementations like Deep Deterministic Policy Gradient (DDPG) or Soft Actor-Critic (SAC).

• Experience Replay: A buffer that stores past transitions (s, a, r, s'). During training, mini-batches are sampled randomly from this buffer to decorrelate sequential experiences, leading to more stable, i.i.d.-like learning.
• Target Networks: Separate, slowly-updated copies of the Actor and Critic networks used to calculate the TD target (r + γ * V_target(s')). This prevents a moving target problem, where the value network being updated is also used to calculate its own target, which can cause divergence.
• Impact: These techniques are largely responsible for enabling Actor-Critic methods to scale to complex problems with deep neural networks.

The Actor-Critic architecture explicitly separates the policy and value functions into two distinct components. The actor is a parameterized policy, typically a neural network, that maps environment states to probability distributions over actions. The critic is a value function, also a neural network, that estimates the expected cumulative reward (the value) of being in a given state. The critic's evaluation is used as a baseline to reduce variance in the actor's policy updates, leading to more stable training than pure policy gradient methods like REINFORCE.

Advantage Actor-Critic is a foundational variant that refines the learning signal. Instead of using the raw value from the critic, it calculates the advantage function: A(s,a) = Q(s,a) - V(s). This represents how much better a specific action is compared to the average action in that state. Key implementations include:

• A2C (Synchronous): A centralized approach where a single learner updates a global network using gradients from multiple parallel environments.
• A3C (Asynchronous): A decentralized, more scalable predecessor where multiple actor-learners asynchronously update a shared global network, though it has largely been superseded by A2C for stability.

Proximal Policy Optimization is a highly popular and robust Actor-Critic algorithm. It introduces a clipped surrogate objective function to prevent excessively large policy updates that can collapse performance. The core mechanism involves:

• Collecting a batch of trajectories using the current policy.
• Computing probability ratios between the new and old policies for each action.
• Clipping these ratios to constrain the update, ensuring the new policy does not stray too far from the old one.
• Optimizing this clipped objective via stochastic gradient descent. PPO's simplicity, sample efficiency, and reliability have made it a default choice for complex RL applications from robotics to game playing.

Soft Actor-Critic is a state-of-the-art off-policy algorithm designed for maximum entropy reinforcement learning. It aims to maximize both expected reward and policy entropy, encouraging exploration and robustness. Key features include:

• An off-policy architecture, allowing reuse of past experience stored in a replay buffer for greater sample efficiency.
• Automatic temperature tuning to balance the reward and entropy terms.
• Separate neural networks for the actor (policy), two Q-function critics (to mitigate overestimation bias), and a value function. SAC excels in continuous control tasks, such as robotic manipulation and locomotion, due to its stability and high performance.

Twin Delayed DDPG is a robust off-policy Actor-Critic algorithm for continuous action spaces, built as an improvement over Deep Deterministic Policy Gradient (DDPG). It addresses DDPG's tendency towards overestimating Q-values through three key techniques:

• Twin Critics: Training two Q-networks and using the minimum of their estimates for the value target, reducing overestimation bias.
• Target Policy Smoothing: Adding noise to the target action to smooth out Q-values and prevent policy exploitation of Q-function errors.
• Delayed Policy Updates: Updating the actor (policy) less frequently than the critics to allow the value estimates to stabilize first. TD3 is known for its reliability in complex physical simulation benchmarks.

Actor-Critic methods are pivotal in training autonomous systems for real-world physical tasks. Their ability to handle high-dimensional, continuous state and action spaces makes them ideal for:

• Robotic Locomotion: Teaching legged robots (e.g., ANYmal, Spot) to walk, run, and recover from pushes.
• Manipulation: Training robotic arms for precise tasks like grasping, assembly, and tool use.
• Autonomous Vehicles: Developing nuanced control policies for steering, acceleration, and braking in simulators.
• Industrial Control: Optimizing complex processes in manufacturing and energy systems. Algorithms like PPO and SAC are frequently deployed in these domains due to their sample efficiency and stability during training in simulation, prior to sim-to-real transfer.

Reinforcement Learning is the overarching machine learning paradigm where an agent learns to make sequential decisions by interacting with an environment to maximize cumulative reward. Actor-Critic is a specific architecture within RL that combines two components:

• Policy-based methods (like the Actor) directly learn the action-selection policy.
• Value-based methods (like the Critic) learn to estimate the value of states or actions. This hybrid approach aims to balance the sample efficiency of value-based methods with the stability and convergence benefits of policy-based methods.

Policy Gradient methods are a core class of RL algorithms that optimize a parameterized policy function (the Actor) directly by ascending the gradient of expected reward. Actor-Critic methods are a subset of policy gradient algorithms that use a value function (the Critic) to reduce the variance of these gradient estimates.

• REINFORCE is a classic policy gradient algorithm that does not use a critic, leading to high-variance updates.
• Advantage Actor-Critic (A2C/A3C) uses the critic to estimate the advantage function (how much better an action is than average), which provides a lower-variance, more stable learning signal for the actor.

Temporal Difference Learning is a central concept in value-based RL where an agent learns by bootstrapping—updating its value estimates based on other estimates. The Critic in an Actor-Critic architecture is typically trained using TD methods.

• The critic's role is to estimate the value function (e.g., state-value V(s) or action-value Q(s,a)).
• TD Error is the difference between the predicted value and the target value (reward plus discounted future value). This scalar TD error is the primary signal used to critique and improve the actor's policy.

Proximal Policy Optimization is a modern, state-of-the-art policy gradient algorithm that uses an Actor-Critic architecture. It introduces a clipped objective function to prevent excessively large policy updates, ensuring stable and reliable training.

• PPO maintains two networks: an Actor (policy) and a Critic (value function).
• The critic estimates the value of states to compute an advantage function.
• The actor's update is constrained to a trust region, making it a robust and widely adopted algorithm for complex environments, from video games to robotic control.

Deep Q-Network is a seminal value-based RL algorithm that uses a deep neural network to approximate the Q-value function. It contrasts with Actor-Critic methods, which explicitly separate policy and value functions.

• DQN is purely value-based: it learns a Q-function and derives a policy by selecting the action with the highest Q-value.
• Actor-Critic is policy-based: the actor directly parameterizes the policy, which can be beneficial in continuous action spaces where finding the max over a Q-function is computationally difficult.
• DQN inspired many advancements later incorporated into Actor-Critic methods, like experience replay and target networks.

Multi-Agent Reinforcement Learning extends RL to environments with multiple interacting agents. Actor-Critic architectures are frequently adapted for MARL, where coordination and competition add complexity.

• In MARL, each agent may have its own actor and critic (Independent Learner).
• More advanced approaches use a centralized critic that has access to global state information to guide decentralized actors, a paradigm known as Centralized Training with Decentralized Execution (CTDE).
• This is highly relevant to Heterogeneous Fleet Orchestration, where multiple autonomous agents (robots, vehicles) must learn coordinated routing and task-allocation policies.