Deep Q-Network (DQN)

A biologically inspired mechanism that breaks the temporal correlations inherent in sequential RF data. Instead of learning from consecutive spectrum transitions, the agent stores its experiences—defined as tuples of (state, action, reward, next state)—in a replay memory buffer. During training, mini-batches are sampled uniformly at random from this buffer.

• Decorrelation: Random sampling breaks the harmful correlation between consecutive samples, satisfying the i.i.d. assumption of stochastic gradient descent.
• Sample Efficiency: Each spectrum interaction can be reused multiple times for weight updates, critical when over-the-air data collection is expensive.
• Stabilization: Prevents the policy from oscillating or diverging by smoothing the data distribution over many past behaviors.

A separate neural network with an identical architecture to the primary Q-network, used to compute the Temporal Difference (TD) target during training. The target network's weights are frozen and only periodically copied from the primary network.

• Removes the Moving Target Problem: Without a target network, the Q-values used to compute the TD error are generated by the same network being updated, creating a harmful feedback loop.
• Stable Regression: By holding the target values fixed for several thousand steps, the learning problem is temporarily reduced to a standard supervised regression task.
• Periodic Hard Update: The target network weights are synchronized with the primary network every C steps, introducing a controlled lag that dampens oscillations.

DQN replaces the traditional tabular Q-table with a deep convolutional neural network (CNN) that maps raw high-dimensional state representations directly to action-value estimates. For cognitive radio, the input is often a spectrogram or a history of spectrum occupancy maps.

• Generalization: The CNN learns to extract relevant features (e.g., interference patterns, signal bandwidths) and can generalize Q-value estimates to previously unseen spectrum states.
• End-to-End Learning: Eliminates the need for hand-crafted feature engineering, learning optimal representations directly from raw IQ data or time-frequency plots.
• Scalability: Handles continuous or extremely large discrete state spaces where a lookup table would be computationally infeasible.

A practical engineering technique to constrain the scale of gradients and prevent numerical instability during training. All positive rewards are clipped to +1 and all negative rewards to -1.

• Uniform Scale: Ensures that the magnitude of the TD error does not vary wildly across different games or tasks, allowing a single set of hyperparameters to work robustly.
• Gradient Clipping: By bounding the error signal, the technique prevents a single anomalous transition from generating an extremely large gradient update that could catastrophically destabilize the network weights.
• Limitation Awareness: While effective for stability, reward clipping discards magnitude information, which can be problematic for spectrum access tasks where maximizing throughput requires precise reward scaling.

The action selection strategy used by DQN to balance the exploration-exploitation tradeoff. With probability ε, the agent selects a random action (exploration); otherwise, it selects the action with the highest estimated Q-value (exploitation).

• Annealing Schedule: The value of ε is typically initialized at 1.0 and linearly or exponentially decayed to a small final value (e.g., 0.1 or 0.01) over millions of frames.
• Spectrum Application: In a cognitive radio context, exploration means occasionally transmitting on a random frequency channel to discover new spectrum opportunities, while exploitation means sticking to known idle channels.
• Simplicity: Despite its simplicity, epsilon-greedy remains a robust baseline against more sophisticated methods like Boltzmann exploration or Thompson sampling.

A critical refinement to the original DQN that addresses the overestimation bias of Q-values. Standard DQN uses the same network to both select and evaluate the best action in the TD target calculation, leading to a systematic upward bias.

• Decoupled Selection & Evaluation: Double DQN uses the primary network to select the best action but the target network to evaluate its value.
• Formula Change: The target becomes R + γ * Q_target(s', argmax_a Q_primary(s', a)).
• Improved Stability: This simple modification leads to more accurate value estimates and significantly better policy performance in noisy environments like dynamic spectrum access.

A model-free reinforcement learning algorithm that learns the value of taking a specific action in a given state. It iteratively updates a Q-table using the Bellman equation: Q(s,a) ← Q(s,a) + α[r + γ max Q(s',a') - Q(s,a)]. The DQN directly replaces this discrete table with a deep neural network to handle high-dimensional state spaces like raw spectrograms, where enumerating every state-action pair is computationally impossible.

A biologically-inspired mechanism that stores the agent's experiences as tuples (state, action, reward, next_state) in a replay memory buffer. During training, the DQN samples random mini-batches from this buffer rather than learning from sequential transitions. This breaks the harmful temporal correlations between consecutive samples and allows the network to reuse rare, high-value experiences multiple times, dramatically stabilizing the learning process in non-stationary spectrum environments.

A separate neural network with an identical architecture to the primary Q-network, but with its parameters frozen and periodically copied. It is used to compute the temporal difference target r + γ max Q_target(s',a'). By decoupling the action-value estimation from the target calculation, this technique prevents the harmful feedback loop where the network chases a constantly shifting target, a phenomenon known as policy oscillation that is particularly severe in dynamic spectrum access scenarios.

The formal mathematical framework underlying DQN-based spectrum access. An MDP is defined by a tuple (S, A, P, R, γ) where:

• S: State space (e.g., channel occupancy vector, sensed interference power)
• A: Action space (e.g., transmit on channel 5, switch to channel 12, stay idle)
• P: Transition probability P(s'|s,a)
• R: Reward function (e.g., +1 for successful packet delivery, -10 for collision with primary user)
• γ: Discount factor weighting future rewards

The fundamental dilemma in cognitive radio learning where the DQN agent must balance exploiting known high-quality channels against exploring potentially superior but untested frequencies. The standard implementation uses an epsilon-greedy strategy: with probability ε, a random action is selected, and with probability 1-ε, the action with the highest Q-value is chosen. Epsilon is typically annealed from 1.0 to a small value like 0.01 over training, ensuring the agent transitions from exploration to exploitation as its knowledge matures.

A more realistic extension of the MDP for spectrum access where the DQN agent cannot directly observe the true environmental state. Instead, it receives noisy observations (e.g., energy detection corrupted by fading) and must maintain an internal belief state. To handle this partial observability, DQN implementations often stack the last k frames of spectrum sensing data as input, providing the convolutional layers with a temporal window that implicitly captures the hidden dynamics of primary user traffic patterns.