Deep Q-Network (DQN)

A finite memory cache that stores the agent's transition tuples (s, a, r, s') from sequential interactions with the RF environment. Instead of learning from consecutive, highly correlated spectrum observations, the agent samples random mini-batches from this buffer for training.

• Decorrelation: Breaks the temporal correlations inherent in time-series spectrum occupancy data, satisfying the i.i.d. assumption of stochastic gradient descent.
• Data Efficiency: Allows rare but critical events—such as the sudden appearance of a primary user—to be replayed multiple times, preventing the network from forgetting low-probability interference scenarios.
• Implementation: Typically implemented as a circular buffer holding the last 1 million transitions, with uniform random sampling to ensure diverse mini-batch composition.

A secondary deep neural network with an identical architecture to the primary Q-network but with frozen parameters. It provides stable temporal difference targets during the Bellman update, decoupling the action selection from the target value computation.

• Stability Mechanism: The target network's weights are updated periodically (e.g., every 10,000 steps) via a hard copy from the online network, preventing harmful feedback loops where the policy chases a constantly shifting target.
• Bellman Equation: The loss is computed as the mean squared error between the online Q-value and the target: r + γ max_a' Q_target(s', a').
• Spectrum Relevance: In dynamic spectrum access, this prevents the agent from developing oscillatory channel selection policies when the occupancy pattern is non-stationary.

The front-end of the DQN that processes raw high-dimensional input—originally video pixels, but adapted for RF as spectrograms or IQ constellations—into a compact latent state representation.

• Input Adaptation: For spectrum access, the input is typically a 2D spectrogram (frequency × time) or a waterfall display, where convolutional layers detect localized energy patterns and temporal-spectral signatures of specific modulation schemes.
• Hierarchical Features: Early layers detect edges and energy bursts; deeper layers combine these into abstract features representing channel occupancy patterns and primary user transmission behaviors.
• Dimensionality Reduction: Compresses a raw 100×100 spectrogram into a compact feature vector before passing it to fully connected layers for Q-value estimation.

An architectural refinement that splits the Q-network into two separate streams after the convolutional layers: one estimating the state value V(s) and another estimating the advantage A(s, a) for each action. These are recombined to produce the final Q-values.

• Efficiency: The agent can learn which spectrum states are inherently valuable (e.g., low-interference channels) without needing to evaluate every action's outcome in that state.
• Aggregation: Q(s,a) = V(s) + A(s,a) - mean(A(s,a)), ensuring identifiability and improving policy evaluation in states where the choice of action has minimal impact.
• Spectrum Benefit: In idle channels where no primary user is present, the advantage stream learns that any action is acceptable, while the value stream focuses on predicting future occupancy.

A critical modification that addresses the overestimation bias inherent in standard DQN by decoupling action selection from action evaluation using the dual-network architecture already present.

• Mechanism: The online network selects the best action *a = argmax_a Q_online(s', a), but the target network evaluates its value: *Q_target(s', a). This prevents the maximization bias where noisy estimates are systematically selected and inflated.
• Overestimation Impact: In spectrum access, overestimating the Q-value of a risky channel can lead to excessive interference with primary users. Double DQN provides conservative, reliable value estimates.
• Adoption: Considered a standard baseline for any DQN-based cognitive radio agent due to its minimal computational overhead and significant stability improvement.

An enhancement to uniform experience replay that assigns sampling probabilities proportional to the temporal difference error magnitude, ensuring that surprising or high-learning-potential transitions are replayed more frequently.

• TD-Error Priority: Transitions where the agent's prediction was significantly wrong—such as an unexpected primary user arrival—receive higher priority, accelerating learning on critical spectrum events.
• Stochastic Sampling: Uses importance-sampling weights to correct the bias introduced by non-uniform sampling, ensuring the update remains unbiased in expectation.
• Spectrum Application: Prioritizes rare interference events and channel-switching transitions, preventing the agent from overfitting to the dominant idle-channel experiences that constitute the majority of the replay buffer.

A biologically inspired mechanism that stabilizes deep reinforcement learning by breaking temporal correlations in sequential data. Instead of learning from consecutive transitions, the agent stores tuples of (state, action, reward, next state) in a replay buffer and samples random mini-batches for training.

• Uniform random sampling reduces variance by preventing the network from overfitting to recent, correlated experiences.
• Prioritized Experience Replay extends this by sampling transitions with higher temporal-difference error more frequently, accelerating learning on surprising or high-value spectrum occupancy events.
• In spectrum access, this prevents the DQN from forgetting rare primary user appearances that occur infrequently in the data stream.

A frozen copy of the primary Q-network used to compute stable temporal-difference targets during training. Without this architectural innovation, the DQN suffers from a harmful feedback loop where the network chases a non-stationary target that shifts with every weight update.

• The target network parameters are periodically updated (hard update) or slowly blended (soft update) from the online network.
• This decoupling prevents catastrophic divergence in value estimates, which is critical when learning spectrum access policies where channel quality fluctuates rapidly.
• Typical update frequencies range from every 1,000 to 10,000 training steps, depending on environment non-stationarity.

An algorithmic refinement that addresses the overestimation bias inherent in standard Q-learning. The max operator used to select the best next action systematically inflates value estimates because the same network both selects and evaluates actions.

• Double DQN decouples action selection from action evaluation by using the online network to choose the best action and the target network to estimate its value.
• This simple modification produces more accurate value functions and improves policy quality in stochastic spectrum environments where channel availability is probabilistic.
• Empirically, Double DQN reduces the overestimation error by up to an order of magnitude compared to vanilla DQN.

A neural network architecture that separates the estimation of state value and action advantage into two distinct streams before combining them at the output layer. This factorization allows the agent to learn which states are valuable without needing to evaluate every action.

• The value stream estimates V(s): how good it is to be in a particular spectrum state regardless of action.
• The advantage stream estimates A(s,a): the relative benefit of taking a specific action compared to the average.
• In spectrum access, this means the agent can recognize that a channel occupied by a primary user is a poor state without needing to learn that every possible action there is bad.

The simplest exploration strategy used with DQNs, where the agent selects a random action with probability ε and the greedy action with probability 1-ε. Despite its simplicity, it remains effective when combined with proper decay schedules.

• Linear decay reduces ε from 1.0 to a small final value (e.g., 0.01) over a fixed number of steps.
• Exponential decay applies a multiplicative factor per episode, enabling faster convergence in stationary spectrum environments.
• For dynamic spectrum access, insufficient exploration causes the agent to miss newly available channels, while excessive exploration causes harmful interference to primary users.

The learning signal that drives all value-based reinforcement learning, measuring the discrepancy between the current Q-value estimate and a better estimate formed from the immediate reward plus the discounted value of the next state.

• TD Error = r + γ max Q(s', a') - Q(s, a), where γ is the discount factor controlling the agent's planning horizon.
• In spectrum access, a high TD error occurs when the agent unexpectedly encounters a primary user on a channel it believed was vacant, triggering a significant value update.
• The magnitude of TD error is used in Prioritized Experience Replay to determine which transitions are most informative for learning.