Model-Free Reinforcement Learning

Unlike model-based methods that attempt to learn a predictive model of the RF environment's state transitions, model-free algorithms treat the environment as a black box. The agent learns solely from sequences of (state, action, reward, next_state) tuples. This eliminates the compounding errors that arise from imperfect environment models, which is critical in volatile wireless channels where accurate modeling of fading, interference, and primary user behavior is computationally intractable.

Model-free methods fall into two primary categories:

• Value-Based: Algorithms like Deep Q-Networks (DQN) learn the optimal action-value function Q*(s,a), which estimates the expected cumulative reward of taking an action in a given state. The policy is derived implicitly by selecting the action with the highest Q-value.
• Policy-Based: Algorithms like Proximal Policy Optimization (PPO) directly parameterize and optimize a stochastic policy π(a|s) without an intermediate value function, enabling natural exploration in continuous action spaces like power control.

The agent discovers optimal spectrum access strategies through iterative interaction rather than prior planning. It must autonomously navigate the exploration-exploitation tradeoff: trying suboptimal channels to gather information versus exploiting known high-quality channels. Techniques like epsilon-greedy (selecting a random action with probability ε) and Thompson Sampling (sampling from posterior reward distributions) are used to ensure the agent does not converge prematurely to a locally optimal but globally poor channel selection policy.

Model-free algorithms typically require a large number of interactions to converge to an optimal policy because they do not leverage a model to simulate hypothetical experiences. In cognitive radio, this translates to a longer learning period during which the secondary user may cause interference or experience poor throughput. Techniques like experience replay—storing past transitions in a buffer and sampling random mini-batches for training—break the temporal correlation between consecutive samples and improve data efficiency, stabilizing training for Deep Q-Networks.

Wireless environments are inherently non-stationary: primary user traffic patterns shift, jammers adapt their strategies, and channel fading statistics drift over time. Model-free methods, particularly policy gradient algorithms like Actor-Critic, can continuously adapt their policy in response to changing reward distributions without needing to detect and relearn a new environment model. This makes them suitable for anti-jamming and dynamic spectrum access scenarios where the adversary or environment is constantly evolving.

To handle the high-dimensional, continuous state spaces common in cognitive radio—such as spectrograms, channel state information vectors, or geolocation coordinates—model-free algorithms rely on deep neural networks as function approximators. A Deep Q-Network uses a convolutional or fully connected network to map raw spectrum observations to Q-values for each possible action (e.g., channel selection, modulation scheme). This eliminates the need for manual feature engineering and allows the agent to learn hierarchical representations directly from raw IQ data or spectrum sensing outputs.

A secondary user employs Q-learning to learn which frequency channel maximizes throughput while avoiding collisions. The agent maintains a Q-table mapping channel indices to expected rewards, updated via the Bellman equation after each transmission attempt.

• State: Current channel occupancy vector from spectrum sensing
• Action: Select channel k for next transmission
• Reward: +1 for successful transmission, -10 for collision with primary user
• Convergence: Typically within 1000-5000 iterations in stationary environments

A Deep Q-Network (DQN) learns to evade a reactive jammer by predicting jammer behavior from spectral observations. The neural network processes a spectrogram of recent channel activity and outputs Q-values for each available frequency.

• Input: Stacked spectrogram frames (84x84x4) representing recent RF activity
• Output: Q-values for each available frequency channel
• Key advantage: Learns jammer patterns without explicit jammer modeling
• Experience replay stabilizes training against non-stationary adversarial behavior

Proximal Policy Optimization (PPO) enables seamless spectrum handoff when a primary user returns. The policy network outputs a probability distribution over target channels, constrained by a trust region to prevent destructive policy updates during critical handoff events.

• Observation space: Current channel SNR, primary user detection probability, queue length
• Action space: Stay on current channel or handoff to one of N backup channels
• Advantage function reduces variance in gradient estimates
• Achieves <50ms handoff latency in trained deployments

Multiple secondary users deploy independent Q-learners without explicit coordination. Each agent treats other secondary users as part of the environment, learning implicit cooperation through shared collision penalties.

• Challenge: Environment non-stationarity from concurrent learning agents
• Mitigation: Optimistic initialization and decaying exploration rates
• Emergent behavior: Agents self-organize onto orthogonal channels
• Scales to 10+ agents without centralized fusion center overhead

An actor-critic architecture jointly optimizes transmission power and modulation-coding scheme selection. The actor network proposes continuous power levels and discrete MCS indices, while the critic estimates the expected spectral efficiency.

• Continuous action space: Transmission power in dBm (actor outputs mean and variance)
• Discrete action space: MCS index selected via Gumbel-Softmax reparameterization
• Critic network: Estimates value function to compute advantage for policy gradient
• Reward: Spectral efficiency minus penalty for exceeding interference temperature limit

Thompson Sampling provides rapid channel ranking when a cognitive radio first powers on in an unknown environment. The algorithm maintains a Beta distribution over success probability for each channel and samples from the posterior to select the next channel.

• Prior: Beta(1,1) uniform prior over success probability per channel
• Update: Increment alpha on success, beta on collision
• Regret bound: Logarithmic regret, provably near-optimal for stationary bandits
• Practical advantage: No hyperparameter tuning required vs. epsilon-greedy

The formal mathematical framework underlying model-free RL. An MDP defines the environment as a tuple (S, A, P, R): a set of states (e.g., channel occupancy), a set of actions (e.g., transmit or switch frequency), transition probabilities defining how actions change the state, and a reward function providing scalar feedback. Model-free algorithms solve MDPs without knowing P—the transition dynamics—by learning purely from sampled experience. This makes them ideal for wireless environments where accurate channel modeling is intractable.

The central dilemma in online learning that model-free agents must resolve. Exploitation means selecting the channel or transmission power known to yield the highest throughput based on current knowledge. Exploration means trying a potentially suboptimal action—like an untested frequency—to gather new information that might reveal a better long-term strategy. Algorithms like epsilon-greedy, Upper Confidence Bound (UCB), and Thompson Sampling provide principled mechanisms to balance this tradeoff, directly impacting the regret and convergence speed of a cognitive radio agent.

A foundational off-policy model-free algorithm that learns the optimal action-value function *Q(s, a)**—the expected cumulative reward of taking action a in state s and following the optimal policy thereafter. The agent iteratively updates Q-values using the Bellman optimality equation without needing a transition model. In cognitive radio, Q-learning enables a secondary user to autonomously discover the best channel access strategy purely through trial-and-error interaction, learning to avoid collisions with primary users by observing experienced interference as negative rewards.

An extension of Q-learning that uses a deep neural network as a function approximator to estimate Q-values in high-dimensional state spaces. Key innovations include experience replay—storing past transitions in a buffer and sampling random mini-batches to break temporal correlation—and target networks that stabilize training. For spectrum access, a DQN can process raw spectrograms or multi-dimensional channel state information as input, outputting Q-values for each possible action (e.g., which frequency to occupy), enabling decision-making in complex, partially observable radio environments.

A policy gradient model-free algorithm that directly optimizes a parameterized policy π_θ(a|s)—the probability of taking action a in state s. PPO constrains policy updates to a trust region using a clipped surrogate objective, preventing destructively large updates that cause performance collapse. This stability makes PPO well-suited for continuous control tasks in cognitive radio, such as fine-grained power control or adaptive modulation selection, where smooth, reliable policy improvement is critical for maintaining link quality during learning.

An extension of the MDP framework where the agent cannot directly observe the true environmental state—a realistic condition in wireless systems. A cognitive radio receives only noisy, incomplete observations (e.g., RSSI samples affected by fading and hidden nodes). The agent must maintain a belief state—a probability distribution over possible true states—and learn policies that map beliefs to actions. Model-free approaches to POMDPs often use recurrent neural networks (e.g., LSTMs) within the policy network to implicitly track state history and infer hidden context from observation sequences.