Exploration-Exploitation Tradeoff

The simplest approach where the agent selects the best-known action with probability 1-ε and explores a random action with probability ε. In cognitive radio, this translates to using the historically best channel most of the time while occasionally sampling other frequencies.

• Fixed ε: Constant exploration rate, simple but suboptimal
• Decaying ε: Gradually reduces exploration as learning progresses
• Drawback: Explores uniformly, wasting time on clearly poor channels
• Use case: Baseline for dynamic spectrum access when computational simplicity is critical

A deterministic algorithm that selects actions by maximizing an optimistic estimate of expected reward. The UCB formula adds an exploration bonus proportional to the uncertainty of each action's value estimate.

• Principle: "Optimism in the face of uncertainty"
• Formula: Select action maximizing Q(a) + c * sqrt(ln(t) / N(a))
• Advantage: Systematically explores uncertain options without randomness
• Spectrum application: Naturally prioritizes rarely-sensed channels that might contain primary user opportunities
• Limitation: Assumes stationary reward distributions, challenging in non-stationary RF environments

A probabilistic Bayesian approach that maintains a posterior distribution over each action's expected reward. At each decision step, the agent samples from these distributions and selects the action with the highest sampled value.

• Mechanism: Naturally balances exploration through posterior uncertainty
• Prior distribution: Typically Beta distribution for binary rewards (channel free/busy)
• Key advantage: Outperforms UCB in practice for many problems with delayed feedback
• Cognitive radio fit: Excels when channel occupancy follows predictable statistical patterns
• Implementation: Requires maintaining and updating belief distributions for each frequency band

Selects actions probabilistically based on their estimated values, using a temperature parameter τ to control exploration intensity. Higher temperatures produce more uniform random selection; lower temperatures increasingly favor the best-known action.

• Formula: Probability of action a ∝ exp(Q(a) / τ)
• Annealing: Gradually decreasing τ shifts from exploration to exploitation
• Advantage: Explores proportionally to value estimates, not uniformly like epsilon-greedy
• Spectrum use: Appropriate when channel quality varies on a gradient rather than binary good/bad
• Challenge: Requires careful temperature scheduling for non-stationary wireless environments

Extends multi-armed bandits by incorporating side information (context) before each decision. In cognitive radio, context includes SNR measurements, time of day, geolocation, or spectrum occupancy history.

• Architecture: Neural network maps context features to predicted rewards per action
• Exploration mechanism: Often uses epsilon-greedy or Thompson sampling on top of predictions
• Key benefit: Generalizes across similar contexts, dramatically accelerating learning
• Example: Learning that certain frequencies are reliably free during nighttime hours
• Implementation: Requires feature engineering of RF environment state into a context vector

Advanced exploration strategy where the agent receives intrinsic rewards for visiting novel or unpredictable states, supplementing the extrinsic reward from successful transmission. This drives systematic exploration of the RF environment.

• Prediction error: Reward proportional to how surprising the observed spectrum state is
• Count-based: Bonus for visiting rarely-seen frequency bands or occupancy patterns
• Key advantage: Explores intelligently even when extrinsic rewards are sparse
• Spectrum application: Discovers intermittent interference patterns or hidden primary user behaviors
• Tradeoff: Adds computational overhead for maintaining state visitation models

A probabilistic algorithm for the multi-armed bandit problem that selects actions based on their posterior probability of being optimal. It maintains a beta distribution over each channel's success rate and samples from these distributions to decide.

• Bayesian approach: Updates beliefs as new rewards are observed
• Natural balance: Probability of selection proportional to probability of optimality
• Spectrum application: Excels in non-stationary RF environments where channel quality drifts over time

A deterministic algorithm that selects actions by maximizing an optimistic estimate of expected reward. It adds an exploration bonus proportional to the uncertainty of each option.

• Formula: Select arm with max (mean_reward + sqrt(2*ln(t)/n_i))
• Exploration bonus: Decreases as an arm is pulled more frequently
• Deterministic policy: Unlike Thompson Sampling, no randomness in selection
• Regret bounds: Provides theoretical guarantees on logarithmic regret growth

A model-free reinforcement learning algorithm that learns the optimal action-selection policy by iteratively updating state-action values (Q-values) based on experienced rewards. Unlike bandits, it handles sequential decisions where actions influence future states.

• Bellman equation: Q(s,a) ← Q(s,a) + α[r + γ*max(Q(s',a')) - Q(s,a)]
• Off-policy learning: Learns optimal policy while following exploratory behavior
• ε-greedy exploration: Selects random action with probability ε, greedy otherwise
• Spectrum handoff: Models channel switching as sequential state transitions

A neural network architecture that combines Q-learning with deep learning to approximate optimal action-value functions in high-dimensional state spaces. Essential when the RF environment has too many states for tabular methods.

• Experience replay: Stores transitions in a buffer and samples randomly to break correlation
• Target network: Uses a separate frozen network to stabilize training
• Spectrogram inputs: Can process raw spectrum data as state representation
• Application: Enables cognitive radios to learn policies directly from raw IQ samples or spectrograms

A multi-armed bandit variant where the agent observes side information (context) before making a decision. This enables adaptive channel selection based on current environmental features like time of day, location, or recent interference patterns.

• Context vector: Encodes observable environmental state
• Linear UCB: Assumes reward is a linear function of context
• Neural contextual bandits: Uses deep networks to model complex context-reward relationships
• Practical advantage: Outperforms context-free methods when environmental patterns are predictable