Actor-Critic Method

The method physically or logically separates the policy (actor) from the value function (critic) . The actor parameterizes the policy π(a|s) and selects actions, while the critic estimates a value function V(s) or Q(s,a) to evaluate those actions. This decoupling allows the actor to update using a low-variance advantage estimate (A = Q - V) rather than a raw, noisy Monte Carlo return.

Actor-Critic methods sit between pure policy gradients (high variance, low bias) and pure value-based methods (low variance, high bias). The critic introduces some bias through function approximation, but dramatically reduces the variance of the policy gradient estimate. This trade-off is explicitly tunable via the TD(λ) trace or GAE (Generalized Advantage Estimation) parameter, which interpolates between Monte Carlo and one-step TD returns.

The architecture supports both learning paradigms:

• On-policy (e.g., A2C, PPO): The critic evaluates only the current policy's actions, requiring fresh samples after each update. This is stable but sample-inefficient.
• Off-policy (e.g., DDPG, SAC): The critic learns a Q-function for any policy, allowing reuse of past experience from a replay buffer. This is critical for real-world RAN deployments where data collection is expensive.

The critic's core signal is the TD error: δ = r + γV(s') - V(s). This single scalar simultaneously drives both networks:

• The critic minimizes δ² to improve its value estimates.
• The actor uses δ (or the advantage) to scale its policy gradient, reinforcing actions that led to better-than-expected outcomes. This shared error signal creates a tight feedback loop between evaluation and improvement.

Unlike DQN, which requires a discrete action space for its argmax operation, Actor-Critic methods natively handle continuous action spaces. The actor directly outputs parameters of a continuous distribution (e.g., mean and variance of a Gaussian). This is essential for RAN optimization tasks like power control, beamforming tilt adjustment, and fractional frequency reuse factor selection, where actions are continuous-valued.

Modern Actor-Critic algorithms (e.g., SAC) explicitly add an entropy bonus to the policy optimization objective: J(π) = E[∑ r + αH(π(·|s))]. This encourages the actor to maintain stochasticity and avoid premature convergence to suboptimal deterministic policies. In dynamic spectrum access, this ensures the agent continues to probe alternative frequency bands rather than locking onto a locally optimal but globally poor allocation.

A class of reinforcement learning algorithms that directly optimize the policy function by estimating the gradient of expected rewards with respect to policy parameters. Unlike value-based methods that derive a policy implicitly from a value function, policy gradients can naturally handle continuous action spaces and learn stochastic policies. The fundamental challenge is high variance in gradient estimates, which is precisely what the critic addresses through baseline subtraction.

• Directly parameterizes the policy: π(a|s; θ)
• Suffers from high variance without a baseline
• Foundation for REINFORCE, PPO, and TRPO algorithms

The critic's core responsibility—estimating the expected cumulative return from a given state or state-action pair. The state-value function V(s) predicts total future reward starting from state s, while the action-value function Q(s,a) predicts return after taking action a in state s. In deep actor-critic architectures, neural networks approximate these functions, enabling generalization across large or continuous state spaces typical in RAN resource management.

• V(s): Expected return from state s under policy π
• Q(s,a): Expected return from taking action a in state s
• Critic minimizes temporal difference error during learning

A critical concept that quantifies how much better a specific action is compared to the average action in a given state: A(s,a) = Q(s,a) - V(s). Using the advantage instead of raw Q-values in policy updates significantly reduces variance without introducing bias. In RAN optimization, this allows the agent to learn that allocating extra resource blocks to a cell-edge user is substantially more valuable than the baseline expectation, accelerating convergence.

• Positive advantage: action is better than average
• Negative advantage: action is worse than average
• Reduces policy gradient variance dramatically

The learning mechanism that drives critic updates by bootstrapping from current value estimates. Instead of waiting for an episode to complete, TD learning updates value predictions using the difference between successive estimates: δ = r + γV(s') - V(s). This enables online, incremental learning essential for real-time RAN control where episodes may be arbitrarily long and the agent must adapt to changing traffic patterns and user mobility continuously.

• TD(0): Simplest form using one-step lookahead
• TD(λ): Balances bias and variance with eligibility traces
• Enables learning without complete episode rollouts

The mathematical foundation proving that the gradient of expected reward with respect to policy parameters can be estimated without knowledge of the environment's transition dynamics. The theorem states: ∇J(θ) ∝ E[∇log π(a|s;θ) · Qπ(s,a)]. This model-free property is essential for wireless network optimization, where precise channel models and user behavior distributions are unknown or non-stationary.

• Eliminates need for environment dynamics model
• Enables model-free learning in complex RAN environments
• Forms the theoretical basis for all actor-critic variants

A technique that adds an entropy bonus to the policy optimization objective to encourage exploration and prevent premature convergence to suboptimal deterministic policies. The augmented objective becomes: J(θ) + β·H(π(·|s)), where H measures policy randomness and β controls exploration strength. In Soft Actor-Critic (SAC), this is a core design principle, producing robust stochastic policies that adapt gracefully to the non-stationary conditions of dynamic spectrum environments.

• Prevents policy collapse to deterministic actions
• β hyperparameter balances exploration vs exploitation
• Critical for discovering diverse resource allocation strategies