Partially Observable MDP (POMDP)

The core of a POMDP is the belief state, a probability distribution over all possible true channel states. Since the cognitive radio cannot directly observe if a primary user is transmitting, it maintains a vector of probabilities—e.g., [P(idle)=0.7, P(busy)=0.3]—updated recursively via Bayesian inference after each noisy sensor observation. This belief is a sufficient statistic of the entire history of actions and observations, meaning the optimal decision depends only on the current belief, not the full past sequence.

The observation function O(o | s', a) defines the probability of perceiving a specific sensor reading o after taking action a and landing in state s'. In spectrum sensing, this models the imperfections of energy detectors or matched filters:

• Probability of Detection (Pd): Correctly sensing a busy channel.
• Probability of False Alarm (Pfa): Mistakenly sensing an idle channel as busy. These probabilities directly shape the belief update and expose the fundamental tension between missed detections (causing interference) and false alarms (wasting transmission opportunities).

The transition function T(s' | s, a) encodes the stochastic dynamics of the radio environment. It specifies the probability of the channel moving from state s to s' when the agent executes action a. This is typically derived from a Primary User Activity Model, such as a two-state Markov chain with ON/OFF periods. The transition model captures the temporal correlation of spectrum occupancy—if a channel was busy in the last time slot, it is more likely to remain busy—enabling the agent to predict future states beyond its immediate, imperfect observation.

The reward function R(s, a) quantifies the desirability of taking action a in state s. For spectrum mobility, this encodes the operational trade-offs:

• Positive reward: Successfully transmitting a data packet on an idle channel.
• Negative penalty: Colliding with a primary user (a severe cost to enforce non-interference).
• Small negative cost: Switching channels (representing handoff latency and signaling overhead).
• Zero or negative reward: Staying idle when a channel is free (opportunity cost). The exact ratio between these rewards critically shapes whether the learned policy is conservative or aggressive.

A policy π(b) maps the continuous belief state b to an action—sense, transmit, or switch channel. Because the belief space is a high-dimensional continuous simplex, solving for the optimal policy is computationally intractable via exact methods. The value function V*(b) represents the maximum expected cumulative discounted reward achievable from belief b. In modern implementations, this is approximated using Deep Q-Networks or Actor-Critic architectures that ingest the belief vector and output Q-values for each action, enabling real-time spectrum mobility decisions.

The Semi-Empirical POMDP (SEP-POMDP) extends the classical framework by learning the transition and observation models directly from real-world spectrum data rather than assuming pre-defined distributions. This addresses the concept drift problem where primary user traffic patterns evolve over time. By integrating a Gaussian Process or Bayesian neural network to model uncertainty in the learned dynamics, the SEP-POMDP quantifies its own model confidence, allowing the cognitive radio to balance exploration of uncertain channels against exploitation of known idle ones.

A probability distribution over all possible true channel states, representing the cognitive radio's subjective uncertainty. In a POMDP, the agent never knows the exact state but maintains this sufficient statistic of history.

• Updated via Bayesian inference after each noisy observation
• Encapsulates the entire history of actions and observations
• Transforms the POMDP into a continuous-state belief MDP

A foundational statistical model where the true channel occupancy evolves as a Markov chain, but the agent only observes a stochastic emission correlated with the hidden state.

• States: Idle or Busy (hidden)
• Observations: RSSI, energy detection, or matched filter outputs
• Used to compute the belief state via the forward algorithm
• Often serves as the underlying transition model within a POMDP

A particle filter approach for approximating the belief state when exact Bayesian updates are computationally intractable. Represents the posterior as a set of weighted samples (particles).

• Handles non-linear and non-Gaussian observation models
• Particles are propagated through the transition model and re-weighted by observation likelihood
• Essential for high-dimensional or continuous channel state spaces

A model-free reinforcement learning approach that learns an optimal handoff policy directly from interaction, without requiring an explicit transition model. A deep neural network approximates the Q-value function.

• Input: Belief state or raw observation history
• Output: Q-values for actions (stay, switch to channel k, sense)
• Learns to maximize link maintenance probability through experience replay
• Contrasts with POMDPs which require a known or learned model

A stochastic framework defining the temporal behavior of licensed users, forming the transition dynamics of the POMDP. Common models include:

• ON/OFF exponential model: Idle and busy periods are exponentially distributed
• Markov Modulated Poisson Process (MMPP): Captures bursty traffic with a Poisson rate governed by a hidden Markov chain
• Phase-Type distributions: Generalize exponential holding times for complex patterns
• Accuracy of this model directly bounds POMDP policy quality

The specific future lookahead window for which the POMDP policy optimizes expected reward. Directly impacts the feasibility of proactive spectrum handoff.

• Short horizons (1-2 slots): Reactive behavior, lower computational cost
• Long horizons (10+ slots): Enables pre-emptive channel reservation and path planning
• The discount factor in the POMDP controls the effective horizon
• Must be balanced against model uncertainty propagation over time