Actor-Critic Architecture

The core design features two distinct neural networks operating in tandem:

• The Actor: A policy network (π) that maps the observed state to a probability distribution over actions (or a deterministic action in continuous spaces). It is responsible for action selection.
• The Critic: A value network (V or Q) that estimates the expected cumulative reward (the value) of being in a given state or of taking a specific action in a state. It provides a scalar evaluation of the actor's decisions. This separation allows each component to specialize, leading to reduced variance in policy updates compared to pure policy gradient methods.

Actor-critic methods are fundamentally built on Temporal Difference (TD) learning. The critic learns by bootstrapping—updating its value estimate for a state based on the immediate reward and its own estimate for the next state. This is expressed by the TD error: δ = r + γV(s') - V(s) Where r is the reward, γ is the discount factor, V(s) is the critic's value for the current state, and V(s') is its value for the next state. This TD error (δ) is the central signal used to train both networks: it criticizes the actor's action and trains the critic's value function.

A primary advantage over pure REINFORCE-style policy gradients is variance reduction. In REINFORCE, the policy gradient is scaled by the total return from an entire episode, which can have high variance. The actor-critic replaces this noisy total return with the critic's estimate (the TD error or Advantage).

• The TD error provides a lower-variance, immediate critique of a single action.
• This leads to more stable gradient estimates, enabling faster and more reliable convergence, especially in environments with long episodes or continuous action spaces.

The architecture is highly flexible regarding action types:

• Discrete Actions: The actor outputs a probability distribution over a finite set of actions (e.g., using a softmax layer).
• Continuous Actions: The actor typically outputs the parameters of a probability distribution, such as the mean (μ) and standard deviation (σ) of a Gaussian. The action is then sampled from this distribution, enabling fine-grained control. This makes actor-critic methods like Deep Deterministic Policy Gradient (DDPG) and Soft Actor-Critic (SAC) the de facto standard for robotic control tasks involving joint torques and velocities.

Actor-critic algorithms can be designed for different learning regimes:

• On-Policy (e.g., A2C, A3C, PPO): The agent learns from experiences generated by its current policy. This often requires fresh samples after each update. Proximal Policy Optimization (PPO) is a prominent on-policy actor-critic that uses a clipped objective to prevent destructively large policy updates.
• Off-Policy (e.g., DDPG, SAC, TD3): The agent can learn from experiences stored in a replay buffer, generated by an older version of the policy or an exploratory policy. This dramatically improves sample efficiency, a critical concern in robotics where real-world data collection is expensive.

Advanced actor-critic methods often use the Advantage function A(s,a) instead of the raw TD error. The Advantage measures how much better a specific action a is compared to the average action in state s. A(s,a) = Q(s,a) - V(s)

• If A(s,a) > 0, the action is better than average; the policy should be adjusted to increase its probability.
• If A(s,a) < 0, the action is worse than average; its probability should be decreased. Using the Advantage function (Advantage Actor-Critic, or A2C) provides an even more refined learning signal, further reducing variance and accelerating policy improvement by focusing on the relative quality of actions.

The synchronous, deterministic variant of A3C. It operates on the same on-policy, advantage-based principles but with a key structural difference:

• All parallel workers synchronize their gradients after collecting a fixed amount of experience.
• A central coordinator performs a single, averaged gradient update to the global model.
• This synchronous update often leads to more stable and reproducible training than the fully asynchronous A3C, especially when using GPUs that benefit from batched operations. It is conceptually simpler to implement and debug.

Policy gradient methods are a class of reinforcement learning algorithms that directly optimize the parameters of a policy function to maximize expected cumulative reward. Unlike value-based methods (e.g., Q-learning), they parameterize the policy itself (e.g., as a neural network) and adjust its parameters in the direction of higher reward.

• Direct Policy Optimization: They work by estimating the gradient of the performance objective with respect to the policy parameters, often using the REINFORCE algorithm or its variants.
• Advantage: Naturally handle continuous action spaces and stochastic policies.
• Challenge: High variance in gradient estimates, which the Actor-Critic architecture helps mitigate by using a value function (the critic) as a baseline.

Deep Deterministic Policy Gradient is an off-policy, actor-critic algorithm specifically designed for continuous action spaces. It combines insights from DQN with deterministic policy gradients.

• Key Components: Uses an actor network that outputs deterministic actions and a critic network (Q-network) that estimates the action-value. Employs target networks and an experience replay buffer for stability.
• Deterministic Policy: The actor maps states directly to specific actions, rather than a probability distribution, making it efficient for control.
• Use Case: Foundational for early deep RL in robotics, such as training robotic arms for manipulation tasks.

Soft Actor-Critic is an off-policy, maximum entropy actor-critic algorithm that aims to maximize both expected reward and policy entropy. This promotes exploration and leads to more robust policies.

• Maximum Entropy Objective: The agent gets bonus reward for acting randomly (high entropy), encouraging it to explore diverse behaviors while still accomplishing the task.
• Stochastic Policy: SAC learns a stochastic policy, which is often more robust to perturbations than deterministic ones—a critical feature for real-world robotics.
• State of the Art: Considered one of the most sample-efficient and stable algorithms for continuous control benchmarks, making it a popular choice for robotic simulation training.

Proximal Policy Optimization is an on-policy, actor-critic algorithm that uses a clipped surrogate objective function to constrain policy updates, ensuring stable and reliable training.

• Clipped Objective: The core innovation is a objective function that prevents any single update from changing the policy too drastically, which avoids performance collapse.
• On-Policy: It uses data collected from the current policy for updates, then discards it. This can be less sample-efficient than off-policy methods but is often simpler to tune.
• Ubiquity in Robotics: Due to its simplicity and robustness, PPO is widely used for training policies in physics simulators (e.g., OpenAI Gym, Isaac Sim) before sim-to-real transfer.

Trust Region Policy Optimization is a precursor to PPO that more rigorously constrains policy updates to a trust region, guaranteeing monotonic improvement under certain theoretical assumptions.

• KL Divergence Constraint: Instead of clipping, TRPO solves a constrained optimization problem where the new policy's divergence from the old policy (measured by KL divergence) is bounded.
• Theoretically Grounded: Provides stronger guarantees of non-decreasing performance with each update.
• Computational Cost: The constrained optimization is more computationally complex than PPO's clipped objective, making PPO the more commonly used variant in practice.

The Advantage Function, A(s,a), is a central concept in actor-critic methods. It measures how much better a specific action is compared to the average action in a given state.

• Definition: Calculated as A(s,a) = Q(s,a) - V(s). It subtracts the state value V(s) (the critic's output) from the action-value Q(s,a).
• Role in Actor-Critic: The actor uses the advantage estimate to update its policy. A positive advantage indicates the action was better than average, so its probability should be increased. A negative advantage means it was worse.
• Reduces Variance: Using the advantage (instead of raw returns) as a baseline is the key mechanism that reduces the variance of policy gradient estimates, leading to more stable learning.