Reinforcement Learning (RL)

The mathematical bedrock of RL, an MDP formalizes sequential decision-making under uncertainty. It is defined by a tuple (S, A, P, R):

• States (S): The set of possible channel occupancy configurations.
• Actions (A): Choices like 'transmit on channel 5' or 'sense channel 12'.
• Transition Probability (P): The probability of moving from a vacant channel to an occupied one.
• Reward (R): A scalar signal, such as +1 for successful throughput or -10 for causing interference to a Primary User (PU). The agent's goal is to find a policy π that maximizes cumulative discounted reward.

A generalization of the MDP that accurately models the reality of wireless sensing. In a POMDP, the agent cannot directly observe the true spectrum state due to noise, fading, or hidden terminals.

• Belief State: Instead of a true state, the agent maintains a probability distribution over all possible states.
• Observation Function: Maps the true state to a noisy sensor reading (e.g., a spectrogram with a missed detection). Solving a POMDP requires integrating sensing history to disambiguate the environment, making it computationally intensive but essential for robust Dynamic Spectrum Access (DSA).

A seminal value-based RL algorithm that uses a deep neural network to approximate the optimal action-value function Q*(s, a). This allows the agent to handle high-dimensional state spaces like raw spectrogram images or multi-channel occupancy grids.

• Experience Replay: Stores transitions (s, a, r, s') in a buffer and samples random mini-batches to break harmful temporal correlations in training data.
• Target Network: A periodically updated copy of the main network stabilizes learning by preventing the target values from shifting during optimization. DQN enables a Cognitive Radio (CR) to learn effective channel selection directly from raw RF data without manual feature engineering.

A state-of-the-art policy gradient method that has become the default choice for many RL applications due to its reliability and sample efficiency. PPO directly optimizes the policy π(a|s) without computing a value table.

• Clipped Surrogate Objective: The core innovation—PPO limits the size of policy updates to a 'trust region' by clipping the probability ratio, preventing destructively large weight changes.
• Actor-Critic Architecture: Uses two networks: an Actor that selects actions and a Critic that evaluates how good the current state is. PPO is highly effective for learning complex spectrum mobility and anti-jamming strategies in continuous or high-dimensional action spaces.

A simplified RL framework that isolates the exploration-exploitation trade-off. The agent faces a row of slot machines (arms) with unknown payout distributions and must decide which to pull to maximize total reward.

• Contextual Bandit: An extension where the agent observes side information (context) before choosing an arm, analogous to sensing RSSI before selecting a channel.
• Regret Minimization: Algorithms like Upper Confidence Bound (UCB) and Thompson Sampling mathematically balance trying new arms against exploiting the current best. MAB models are computationally lightweight and ideal for rapid, stateless channel selection in Listen-Before-Talk (LBT) scenarios.

Extends RL to environments with multiple coexisting agents, where the dynamics are non-stationary from any single agent's perspective. In wireless networks, every secondary user is an agent, and their transmissions interfere with each other.

• Centralized Training Decentralized Execution (CTDE): The dominant paradigm. During training in a simulator, agents share global state information. During deployment, they act using only local observations.
• Emergent Coordination: MARL agents can learn sophisticated spectrum etiquette and implicit channel sharing protocols without explicit communication, crucial for scalable CBRS-like deployments.