Markov Decision Process (MDP)

The finite set of all possible situations the agent can encounter. In cognitive radio, a state typically encodes channel occupancy, signal-to-noise ratio (SNR), and battery level.

• Fully Observable: Agent sees the true state (MDP).
• Partially Observable: Agent receives noisy observations (POMDP).
• Example: A vector representing 10 channels where 1 is occupied and 0 is idle.

The finite set of all possible moves the agent can execute in a given state. Actions alter the environment and trigger state transitions.

• Discrete Actions: Select channel 5, increase power, or switch modulation.
• Continuous Actions: Adjust transmission power by a specific dBm value.
• In a Multi-Armed Bandit model, the action is simply choosing which frequency arm to pull.

The dynamics model defining the probability of moving from state s to state s' after taking action a. This encodes environmental stochasticity.

• Deterministic: Action always leads to the same next state.
• Stochastic: Action has a probability distribution over outcomes.
• In Model-Free RL (like Q-Learning), the agent learns optimal behavior without explicitly knowing these probabilities.

The scalar feedback signal defining the immediate goal. The agent's objective is to maximize cumulative reward over time.

• Positive Reward: Successful packet transmission, high throughput.
• Negative Reward: Collision with a primary user, high latency, or power waste.
• Reward Shaping adds intermediate rewards to accelerate learning in sparse environments where feedback is rare.

The agent's brain—a mapping from states to actions. The policy dictates behavior and is the output of the learning process.

• Deterministic Policy: π(s) = a (always take action a in state s).
• Stochastic Policy: π(a|s) outputs a probability distribution over actions.
• Deep RL algorithms like PPO and Actor-Critic use neural networks to approximate optimal policies in high-dimensional state spaces.

A value between 0 and 1 that determines the present value of future rewards. It balances immediate gratification against long-term planning.

• γ close to 0: Myopic agent, cares only about immediate reward.
• γ close to 1: Farsighted agent, values future rewards almost equally.
• In spectrum access, a high gamma encourages the agent to sacrifice a vacant channel now to secure a better channel later.

The core application of MDPs in cognitive radio involves treating each frequency band as a state with a probability of being idle or occupied. The agent learns a policy that maximizes throughput by selecting channels with the highest expected availability.

• State Space: Binary occupancy vectors for N frequency channels
• Action Space: Switch to channel k or stay on current channel
• Transition Probability: Modeled from historical spectrum sensing data
• Reward Function: +1 for successful packet transmission, -1 for collision with primary user

A Deep Q-Network (DQN) can approximate the optimal action-value function when the state space becomes too large for tabular Q-Learning, enabling real-time decisions across hundreds of channels.

In contested electromagnetic environments, a cognitive radio must evade malicious interference. The jammer's behavior is modeled as part of the MDP's transition dynamics, allowing the radio to learn a counter-policy.

• State: Current frequency, observed jamming signal power, and SINR
• Action: Hop to a new frequency, adjust transmit power, or switch modulation scheme
• Reward: Successful packet delivery ratio minus energy cost

Using Proximal Policy Optimization (PPO) , the radio learns to predict jamming patterns and proactively switch to clean channels before the jammer can lock on. This transforms a reactive defense into a predictive, autonomous countermeasure.

When a primary user reclaims a channel, the secondary user must execute a seamless handoff to maintain connectivity. The MDP framework optimizes this process by considering both current link quality and the cost of switching.

• State: Current channel quality, buffer occupancy, and target channel list
• Action: Initiate handoff to a specific backup channel or wait
• Reward: Negative cost for handoff latency, positive reward for maintaining QoS

The learned policy balances the exploration-exploitation tradeoff—probing new channels for better long-term performance versus staying on a known adequate channel to avoid switching overhead. This is critical for latency-sensitive applications like voice over cognitive radio.

When multiple secondary users compete for the same spectrum, the problem extends to a Stochastic Game, a multi-agent generalization of the MDP. Each radio is an independent agent optimizing its own reward while the environment includes the actions of other radios.

• State: Joint channel occupancy as perceived by each agent
• Action: Transmit on channel k with power level p
• Reward: Individual throughput minus penalty for collisions with peers

Decentralized reinforcement learning algorithms, such as independent Q-learning, allow each radio to converge to a Nash equilibrium without explicit coordination. This enables scalable, infrastructure-free dynamic spectrum access in dense deployments.

For battery-constrained cognitive radios in sensor networks or IoT devices, the MDP framework incorporates energy consumption into the decision process. The agent learns to transmit only when channel conditions and data priority justify the power expenditure.

• State: Battery level, channel quality, data queue length, and energy harvesting rate
• Action: Transmit at full power, transmit at reduced power, or enter sleep mode
• Reward: Data throughput minus a weighted energy cost coefficient

This formulation creates a constrained MDP where the policy maximizes throughput subject to an energy budget. The resulting behavior opportunistically exploits high-quality channels while conserving power during poor conditions, extending operational lifetime without sacrificing critical data delivery.

Real-world spectrum sensing is never perfect—false alarms and missed detections introduce uncertainty. A Partially Observable MDP (POMDP) extends the MDP framework by maintaining a belief state, a probability distribution over possible true channel states.

• Belief State: Updated via Bayesian inference after each sensing observation
• Action: Sense a specific channel, transmit, or remain silent
• Observation: Binary busy/idle reading with known false alarm and missed detection rates

The optimal POMDP policy balances sensing actions to reduce uncertainty against transmission actions to exploit opportunities. This framework explicitly accounts for the hidden node problem and sensing hardware imperfections, producing robust policies that gracefully degrade rather than catastrophically fail under noisy observations.

The fundamental dilemma in reinforcement learning where an agent must balance exploration (trying unfamiliar actions to discover potentially higher rewards) against exploitation (selecting known actions that yield high returns). In spectrum access:

• Exploration: Tuning to an untested frequency channel to measure its quality
• Exploitation: Staying on a known idle channel with high signal-to-noise ratio An ε-greedy strategy resolves this by selecting a random action with probability ε and the greedy action otherwise. Decaying ε over time shifts the agent from exploration to exploitation as knowledge accumulates.