Actor-Critic Model

The Critic generates a scalar TD error that serves as the primary feedback mechanism for updating both networks, replacing the raw Monte Carlo return used in REINFORCE.

• TD Error Formula: δ = r + γV(s') - V(s), where r is the immediate reward, γ is the discount factor, and V(s') is the Critic's estimate of the next state's value.
• Advantage Function: The TD error approximates the advantage A(s,a) = Q(s,a) - V(s), quantifying how much better a specific action was compared to the Critic's baseline expectation.
• Variance Reduction: By subtracting the state baseline V(s), the Actor's policy gradient update uses only the relative advantage, dramatically reducing the variance of gradient estimates without introducing bias.

Actor-Critic methods employ bootstrapping, where the Critic's own estimate of future value is used to update its current estimate, creating a fundamental design tension.

• Bootstrapping Mechanism: The Critic updates V(s) toward the target r + γV(s'), which itself depends on the Critic's potentially inaccurate approximation. This introduces bias but reduces variance compared to waiting for episode termination.
• Bias Introduction: Unlike pure Monte Carlo methods that use complete return sequences, bootstrapping propagates approximation errors backward through the value function, potentially leading to systematic underestimation or overestimation.
• Online Learning: Bootstrapping enables fully online, step-by-step updates without requiring episode completion, making Actor-Critic suitable for continuous cognitive radio tasks like perpetual spectrum monitoring.

The Actor's update rule is a policy gradient theorem implementation where the Critic's value estimate serves as a learned baseline to center the reward signal.

• Update Rule: ∇J(θ) = E[∇log π(a|s) * (Q(s,a) - V(s))]. The term (Q(s,a) - V(s)) is the advantage, indicating whether the action outperformed the baseline.
• Baseline Subtraction: Subtracting V(s) ensures that actions leading to better-than-expected outcomes increase in probability, while worse-than-expected actions decrease, regardless of the absolute reward magnitude.
• Compatibility with Continuous Actions: Unlike value-based methods requiring discrete action maximization, the policy gradient approach naturally handles continuous action spaces such as transmission power levels or frequency tuning parameters.

Actor-Critic architectures bifurcate into on-policy and off-policy families, each with distinct data efficiency and stability characteristics relevant to spectrum access.

• On-Policy (A3C, PPO): The Actor generates trajectories using its current policy π_θ, and these same trajectories train both networks. Requires fresh data after each update, limiting sample efficiency but providing stable convergence.
• Off-Policy (DDPG, SAC): The Actor can learn from trajectories generated by a different behavioral policy, often stored in a replay buffer. This enables data reuse and sample-efficient learning from historical spectrum occupancy logs.
• Entropy Regularization: Off-policy methods like Soft Actor-Critic (SAC) augment the objective with an entropy bonus H(π(·|s)), encouraging exploration by rewarding stochastic policies—critical for discovering underutilized frequency bands.

A class of reinforcement learning algorithms that directly optimize the policy function without requiring a value function. Unlike value-based methods that derive a policy implicitly, policy gradients learn a parameterized policy that maps states to a probability distribution over actions. The Actor-Critic model is a direct extension designed to solve the high variance problem inherent in pure policy gradient methods like REINFORCE. By introducing a critic to estimate the value function, the actor's updates become more stable and sample-efficient.

A central prediction method that learns by bootstrapping from the current estimate of the value function. The critic in an Actor-Critic architecture typically uses TD error to evaluate the action taken by the actor. This error signal represents the difference between the predicted reward and the actual reward plus the discounted value of the next state. It serves as a crucial feedback mechanism, telling the actor whether its chosen action was better or worse than expected.

A metric used to determine how much better a specific action is compared to the average action in a given state. It is formally defined as the difference between the Q-value (state-action value) and the V-value (state value). Many advanced Actor-Critic implementations, such as Advantage Actor-Critic (A2C), use the advantage function to update the actor. This reduces variance further by focusing learning on the relative benefit of an action rather than its absolute value.

An off-policy Actor-Critic algorithm designed for continuous action spaces. It combines the deterministic policy gradient approach with the experience replay and target network techniques from Deep Q-Networks (DQN). The actor directly outputs a specific action vector instead of a probability distribution, making it suitable for robotics and autonomous wireless control tasks like fine-tuning transmission power.

A parallel training framework where multiple agent instances interact with independent copies of the environment simultaneously. Each worker computes gradients locally and asynchronously updates a global network. This decorrelates the training data and eliminates the need for an experience replay buffer. The advantage formulation stabilizes learning, making A3C a highly efficient and robust on-policy algorithm for complex cognitive radio decision-making.

An off-policy maximum entropy reinforcement learning algorithm that optimizes a stochastic policy with an entropy bonus. This encourages the agent to succeed at the task while acting as randomly as possible, leading to superior exploration and robustness. SAC is highly sample-efficient and less brittle to hyperparameter settings, making it a state-of-the-art choice for dynamic spectrum access problems where the environment is non-stationary.