Markov Decision Process (MDP)

An extension of the MDP framework that accurately models the sensing uncertainty inherent in wireless systems. Unlike a standard MDP, the agent cannot directly observe the true spectrum state (e.g., whether a primary user is transmitting). Instead, it maintains a belief state—a probability distribution over possible states—updated via noisy observations like spectrum sensing data. This is critical for cognitive radio because missed detections can cause harmful interference, making POMDPs the rigorous mathematical model for robust dynamic spectrum access.

A foundational model-free reinforcement learning algorithm that learns the optimal action-value function without requiring a model of the RF environment. The agent iteratively updates Q-values for state-action pairs using the Bellman equation, bootstrapping from experienced transitions. Key characteristics for spectrum access:

• Off-policy: Learns the optimal policy while following an exploratory behavior policy
• Temporal Difference: Updates estimates based on other learned estimates
• Convergence Guarantees: Converges to the optimal policy given sufficient exploration in tabular settings

Combines Q-learning with deep neural networks to handle high-dimensional state spaces that are impossible to represent in tabular form. A DQN approximates the Q-function using a convolutional or fully connected network, enabling agents to process raw spectrum occupancy grids or spectrograms directly. Two key innovations stabilize training:

• Experience Replay: Stores transitions in a buffer and samples random mini-batches to break temporal correlations
• Target Network: Uses a separate, periodically updated network to compute TD targets, reducing harmful feedback loops

A policy gradient method that has become the default algorithm for continuous control and spectrum access due to its stability and sample efficiency. PPO constrains policy updates using a clipped surrogate objective, preventing destructively large parameter changes that could cause an agent to suddenly interfere with primary users. Benefits for RF applications:

• Handles continuous action spaces (e.g., precise power control, frequency tuning)
• More stable than DQN for non-stationary wireless environments
• Supports multi-agent extensions for distributed spectrum sharing

A simplified decision framework where an agent selects among K independent channels to maximize cumulative reward, modeling the pure exploration-exploitation trade-off without state transitions. Each channel arm provides a stochastic reward (e.g., throughput, SNR). MAB algorithms like Upper Confidence Bound (UCB) and Thompson Sampling are directly applicable to channel selection in spectrum access, offering lightweight alternatives to full MDP formulations when temporal dynamics are negligible.

A category of reinforcement learning where the agent explicitly learns or is provided with a predictive model of environment dynamics—the transition function P(s'|s,a) and reward function R(s,a). In spectrum access, this model can predict how channel occupancy evolves over time. The agent uses the model for planning via simulated rollouts or model predictive control, dramatically improving sample efficiency. This is critical when real-world spectrum interaction is costly or when safe exploration constraints prohibit trial-and-error learning on live networks.