Q-Learning

Q-Learning does not require a model of the environment's transition dynamics or reward function. The agent learns purely from experienced state-action-reward transitions, making it ideal for spectrum access where accurate RF propagation models are difficult to obtain. The algorithm builds its knowledge incrementally through direct interaction with the electromagnetic environment.

Q-Learning is an off-policy algorithm, meaning it learns the optimal policy while following an exploratory behavior policy. It uses temporal difference (TD) learning to update Q-values based on the difference between current estimates and observed rewards plus discounted future values. This allows the agent to learn from hypothetical actions it did not actually take.

The core update rule is based on the Bellman optimality equation:

Q(s,a) ← Q(s,a) + α[r + γ max Q(s',a') - Q(s,a)]

Where:

• α (alpha): Learning rate controlling update magnitude
• γ (gamma): Discount factor for future rewards
• r: Immediate reward received
• max Q(s',a'): Maximum estimated value of the next state

Q-Learning is proven to converge to the optimal action-value function under specific conditions:

• All state-action pairs must be visited infinitely often
• The learning rate α must decay appropriately over time
• The environment must be a finite Markov Decision Process (MDP)

This theoretical guarantee provides confidence when deploying Q-Learning for mission-critical spectrum access decisions.

Q-Learning requires explicit exploration mechanisms to discover optimal policies:

• ε-greedy: Selects a random action with probability ε, otherwise exploits the current best action
• Softmax/Boltzmann: Selects actions probabilistically based on their relative Q-values
• Upper Confidence Bound (UCB): Balances exploration by considering uncertainty in value estimates

In spectrum access, exploration is critical for discovering spectrum holes in dynamic RF environments.

When state spaces become large—such as raw spectrogram data or multi-dimensional channel occupancy maps—tabular Q-Learning becomes infeasible. The Deep Q-Network (DQN) extends Q-Learning by using a neural network to approximate the Q-function, enabling:

• Experience replay to break temporal correlations
• Target networks to stabilize training
• Handling of continuous or high-dimensional state spaces

The mathematical foundation for Q-Learning, modeling sequential decision-making in stochastic environments. An MDP is defined by:

• States (S): Channel occupancy and SNR conditions
• Actions (A): Frequency selection, power adjustment
• Transition Probabilities (P): Likelihood of channel state changes
• Reward Function (R): Throughput achieved minus interference cost Q-Learning solves the MDP without requiring explicit knowledge of P, making it ideal for unknown RF environments.

Extends Q-Learning by using a deep neural network to approximate the Q-function, handling high-dimensional state spaces like raw spectrograms. Key innovations include:

• Experience Replay Buffer: Stores transitions (s, a, r, s') and samples random mini-batches to break temporal correlations
• Target Network: A periodically updated copy of the Q-network that stabilizes training by reducing moving-target effects DQNs enable cognitive radios to learn directly from raw IQ data without manual feature engineering.

The fundamental dilemma in Q-Learning where the agent must balance:

• Exploration: Trying new frequency channels to discover potentially higher-throughput opportunities
• Exploitation: Selecting known high-reward channels to maximize immediate throughput The epsilon-greedy strategy selects a random action with probability ε and the greedy action with probability 1-ε. In spectrum access, decaying ε over time allows the agent to transition from learning the RF environment to optimizing performance.

Extends the MDP framework to account for imperfect spectrum sensing, where the agent cannot directly observe true channel occupancy. Instead, it maintains a belief state—a probability distribution over possible states updated via Bayesian inference. Q-Learning applied to POMDPs requires:

• Recurrent architectures (LSTM, GRU) to integrate observation history
• Memory-augmented policies that infer hidden primary user activity from noisy RSSI measurements This accurately models real-world cognitive radio uncertainty caused by fading and shadowing.

A simplified special case of Q-Learning with a single state and multiple actions, directly applicable to channel selection. The agent treats each frequency as an 'arm' with an unknown reward distribution. Key algorithms include:

• Upper Confidence Bound (UCB): Selects channels based on optimistic reward estimates plus exploration bonuses
• Thompson Sampling: Maintains Bayesian posterior distributions over channel quality and samples from them MABs provide lightweight, analytically tractable solutions for spectrum access when temporal dynamics are minimal.

A policy gradient alternative to Q-Learning that directly optimizes the policy without learning a value function. PPO constrains policy updates using a clipped surrogate objective to prevent destructively large updates. Advantages over Q-Learning for spectrum access:

• Continuous action spaces: Native support for power control and beamforming
• Stable training: Less sensitive to hyperparameter choices than DQN
• Sample efficiency: Reuses trajectories via importance sampling Often preferred for complex multi-channel access with continuous transmission parameters.