Contextual Bandit

Unlike standard multi-armed bandits that rely solely on historical reward averages, a contextual bandit observes a feature vector (the context) before selecting an action. This context encodes the current state of the environment—such as channel occupancy patterns, signal-to-noise ratio (SNR), or geolocation—allowing the agent to predict which arm (e.g., frequency channel) will yield the highest reward under the specific observed conditions. The decision policy maps contexts to actions, enabling personalized or situation-aware optimization.

The relationship between context and expected reward can be modeled with varying complexity:

• Linear Contextual Bandits: Assume the expected reward is a linear function of the context features. Algorithms like LinUCB and LinTS are computationally efficient and provide strong theoretical regret bounds.
• Non-Linear Contextual Bandits: Employ deep neural networks to model complex, non-linear relationships. Neural Bandits and Deep Contextual Bandits use representation learning to handle high-dimensional raw context, such as raw spectrograms, without manual feature engineering.

A core challenge is balancing exploration (trying under-sampled actions to gather data) with exploitation (choosing the current best-known action). Contextual bandit algorithms explicitly manage this trade-off using principled strategies:

• Upper Confidence Bound (UCB): Adds an exploration bonus to actions with high uncertainty.
• Thompson Sampling: Maintains a posterior distribution over model parameters and samples from it to make probabilistic decisions.
• Epsilon-Greedy: Selects a random action with probability ε, otherwise exploits. This balance is critical in cognitive radio to avoid both harmful interference (exploration risk) and spectrum underutilization (excessive caution).

Contextual bandits operate in an online learning setting where the agent learns incrementally from each interaction. Performance is measured by cumulative regret: the difference between the total reward obtained and the reward that an optimal oracle policy would have achieved. The goal is to achieve sub-linear regret, meaning the average per-step regret approaches zero over time. Algorithms like LinUCB provide theoretical guarantees on regret bounds, ensuring the agent converges to near-optimal channel selection without prior knowledge of the environment.

In dynamic spectrum access, a cognitive radio acts as the agent, available frequency channels are the arms, and the context includes real-time environmental features:

• Sensed RSSI values across the band
• Historical occupancy statistics
• Time of day and geolocation
• Modulation classification outputs The reward is typically a function of achieved throughput minus a penalty for collisions with primary users. This allows the radio to learn a policy that selects the best idle channel based on current spectrum conditions without requiring an explicit model of primary user behavior.

Evaluating a contextual bandit policy offline requires counterfactual reasoning because logged data only contains rewards for actions that were actually taken. Techniques like Inverse Propensity Scoring (IPS) re-weight historical samples to estimate the performance of a new policy without deploying it live. This is essential for safely testing updated channel selection strategies against recorded spectrum data before risking real-world interference in operational cognitive radio networks.

A cognitive radio observes context features—such as time of day, geolocation, and recent spectrum occupancy patterns—before selecting a frequency. The contextual bandit algorithm learns a policy mapping from these features to channel rewards, enabling it to predict which band is likely free without exhaustive sensing. Unlike a standard MAB that ignores context, this approach adapts instantly to changing environmental conditions, such as rush-hour interference in urban cells or scheduled radar pulses in military bands.

In dense heterogeneous networks, a secondary transmitter uses contextual bandits to jointly optimize frequency and transmission power. The context vector includes:

• Measured RSSI from neighboring nodes
• CSI feedback from the intended receiver
• Historical ACK/NACK ratios

The algorithm learns to map this context to power levels that maximize throughput while keeping interference below regulatory thresholds, adapting in real-time as neighboring devices join or leave the network.

When facing a reactive jammer, a cognitive radio employs a contextual bandit to select the next hop frequency. The context captures the observed jamming pattern—sweep rate, duty cycle, and targeted bands—over a recent time window. The bandit learns to associate specific jamming signatures with safe frequencies, enabling predictive evasion rather than blind random hopping. This approach maintains link integrity even against intelligent jammers that adapt their strategy.

Cognitive radios querying a TV White Space (TVWS) database face latency and bandwidth costs. A contextual bandit learns when to trust cached spectrum availability data versus requesting fresh updates. The context includes:

• Time since last query
• Device velocity (fast-moving users need fresher data)
• Historical database accuracy for the current location

This minimizes query overhead while ensuring regulatory compliance, critical for battery-constrained IoT sensors operating in shared spectrum.

A multi-mode terminal equipped with LTE, Wi-Fi, and NR-U interfaces uses a contextual bandit to select the optimal radio access technology per packet flow. The context vector encodes application type (voice, video, bulk data), mobility state, and per-interface load metrics. The bandit learns to route latency-sensitive traffic to licensed spectrum while offloading best-effort data to unlicensed bands, maximizing aggregate throughput and quality of service.

In cooperative spectrum sensing, a fusion center uses a contextual bandit to weight the reliability of each secondary user's sensing report. The context includes:

• SNR of the reporting node
• Historical false alarm rate of that specific hardware
• Correlation with other nodes' reports

The bandit learns to discount reports from nodes experiencing deep fades or suspected of being compromised, significantly improving global detection probability in adversarial or fading environments.