Markov Decision Process (MDP)

The finite set of all possible situations the cognitive radio can encounter. A state is a distinct configuration of the environment at a specific time step.

• Spectrum Context: A state might encode the occupancy status of N channels (idle/busy), the current signal-to-noise ratio (SNR), and the remaining battery level.
• Markov Property: The state must capture all relevant history. The probability of transitioning to a future state depends only on the current state and action, not the sequence that preceded it.
• Discretization: Continuous metrics like SNR are typically quantized into discrete bins to maintain a manageable state space size.

The finite set of all possible operations the cognitive radio agent can execute in any given state. The agent's goal is to learn which action to take in which state.

• Spectrum Actions: Examples include SENSE_CHANNEL_5, TRANSMIT_ON_CHANNEL_3, INCREASE_POWER, SWITCH_MODULATION, or STAY_IDLE.
• Policy Dependency: The action space directly constrains the agent's behavior. A larger action space offers more control but increases the complexity of finding an optimal policy.
• Deterministic vs. Stochastic: While the action choice can be probabilistic (e.g., epsilon-greedy), the action set itself is a defined list of discrete operations.

The core stochastic model of the environment, defining the probability of moving from state s to state s' after taking action a. This captures the uncertainty inherent in wireless channels.

• Formal Definition: P(s' | s, a) = Pr{S_{t+1} = s' | S_t = s, A_t = a}.
• RF Dynamics: This matrix encodes the likelihood that a channel becomes occupied by a primary user, the probability of packet loss due to fading, or the chance of successful handoff.
• Model-Based vs. Model-Free: In model-based reinforcement learning, the agent learns this transition function explicitly. In model-free methods like Q-Learning, the agent learns the value of actions without ever modeling P directly.

A scalar feedback signal that defines the immediate goal of the cognitive radio. The agent's sole objective is to maximize the cumulative sum of these rewards over time.

• Design is Critical: The reward function shapes all learned behavior. A poorly designed reward leads to unintended and often destructive policies.
• Spectrum Rewards: Common examples include:
  • +10 for each successfully transmitted packet.
  • -100 for causing interference to a primary user.
  • -1 per time step to penalize latency and encourage spectral efficiency.
  • +5 for maintaining a target bit error rate.
• Immediate vs. Delayed: The function R(s, a, s') provides the immediate reward, but the agent must learn to sacrifice short-term gain for long-term cumulative return.

A parameter between 0 and 1 that determines the present value of future rewards. It mathematically bounds the infinite sum of rewards and controls the agent's farsightedness.

• Mathematical Role: The return G_t = R_{t+1} + γR_{t+2} + γ²R_{t+3} + ...
• γ → 0 (Myopic): The agent is extremely short-sighted, caring only about maximizing the immediate next reward. It will exploit a high-throughput channel even if it risks imminent collision.
• γ → 1 (Farsighted): The agent heavily weights distant future rewards. It will patiently explore or sacrifice a time slot now to secure a long-term, interference-free channel later.
• Convergence: A discount factor less than 1 ensures that the infinite sum of rewards converges to a finite value, making policy evaluation mathematically tractable.

The agent's strategy—a mapping from every possible state to a probability of selecting each action. This is the ultimate output of the MDP solving process.

• Deterministic Policy: π(s) = a. The agent always takes the same action in state s.
• Stochastic Policy: π(a|s) = Pr{A_t = a | S_t = s}. The agent chooses actions probabilistically, which is crucial for exploration and in multi-agent game-theoretic equilibria.
• Optimal Policy (π)**: The policy that maximizes the expected cumulative discounted reward from any starting state. In a cognitive radio, π dictates the perfect sequence of sensing, channel selection, and power control decisions to maximize throughput without causing interference.

An autonomous entity that learns an optimal spectrum access policy through trial-and-error interaction with the RF environment. The agent observes the current state (e.g., channel occupancy), takes an action (e.g., transmit or switch frequency), and receives a scalar reward signal. Over time, it maximizes cumulative reward by mapping states to actions. This is the practical implementation of the MDP framework in a cognitive radio.

A model-free reinforcement learning algorithm that enables a cognitive radio to learn the optimal action-selection policy without requiring a model of the environment's transition probabilities. The agent maintains a Q-table that estimates the expected utility of taking a given action in a given state. It is particularly suited for discrete state and action spaces, such as a finite set of frequency channels, and converges to an optimal policy given sufficient exploration.

The fundamental dilemma in cognitive radio learning where the system must choose between trying new, uncertain frequency bands (exploration) and using the best-known band (exploitation) to maximize long-term reward. An MDP's solution strategy must balance these competing objectives. Common mechanisms include the epsilon-greedy strategy, where the agent selects a random action with probability ε, and Softmax action selection, which varies choice probability by estimated value.

A simplified reinforcement learning model representing a special case of an MDP with a single state. The cognitive radio must repeatedly choose one of several frequency channels (the 'arms'), each providing a stochastic reward (e.g., throughput). The agent faces the pure exploration-exploitation trade-off without the complexity of state transitions. MAB algorithms like Upper Confidence Bound (UCB) and Thompson Sampling provide efficient channel selection strategies.

The process by which a secondary user seamlessly vacates its current frequency channel upon detecting a returning primary user and transitions its ongoing communication to another available spectrum hole. This is a direct consequence of the transition function in the MDP, where the state changes from 'channel free' to 'channel occupied'. The handoff policy must minimize latency and packet loss to maintain quality of service.

A mathematical framework for modeling strategic interactions among multiple independent cognitive radios, extending the single-agent MDP to a multi-agent setting. Each radio's reward depends on the actions of others. The analysis seeks a Nash Equilibrium, a stable state where no single radio can improve its performance by unilaterally changing its transmission strategy. This is formalized as a Stochastic Game, a generalization of the MDP.