Deep Reinforcement Learning Sensing

Unlike static energy detectors, a DRL agent learns a stochastic policy that maps raw spectrum observations directly to sensing actions. The agent maximizes cumulative reward—typically a weighted function of throughput and interference avoidance—by adjusting sensing duration, detection thresholds, and channel selection on a per-time-slot basis. This enables the radio to spend minimal time sensing in quiet bands while increasing vigilance in congested or high-risk channels.

DRL-based sensing operates without an explicit model of the primary user's traffic pattern or channel fading statistics. The agent learns directly from interaction, making it robust to non-stationary environments where PU behavior evolves over time. Techniques like Deep Q-Networks (DQN) and Proximal Policy Optimization (PPO) allow the agent to discover optimal sensing strategies even when the Markov state transition probabilities are unknown.

The reward function is the critical design element. It encodes competing objectives into a scalar signal:

• Positive reward for successful data transmission (throughput maximization)
• Heavy negative penalty for colliding with a primary user (interference minimization)
• Small negative cost for time spent sensing (efficiency pressure) This shaping forces the agent to discover the Pareto-optimal frontier between aggressive spectrum access and conservative protection.

Real-world spectrum sensing is a Partially Observable Markov Decision Process (POMDP)—the agent never sees the true full state. To handle this, DRL sensing agents often incorporate Long Short-Term Memory (LSTM) or Gated Recurrent Unit (GRU) layers within their policy networks. These recurrent structures maintain an internal belief state about PU activity patterns, enabling the agent to infer hidden occupancy trends from noisy, sequential observations without explicit state estimation.

Training a DRL sensing agent directly on live spectrum is impractical due to the risk of causing harmful interference. Instead, agents are trained in high-fidelity simulated environments that model PU traffic patterns, fading channels, and noise uncertainty. Domain randomization—varying simulation parameters like SNR, PU duty cycle, and channel coherence time during training—produces policies that transfer robustly to real hardware without additional fine-tuning.

Traditional systems separate sensing from transmission. DRL unifies them into a single composite action space. At each time step, the agent selects a tuple: (sensing mode, channel index, transmission power, modulation scheme). This joint optimization captures the coupling between sensing quality and communication performance—for example, the agent learns that a brief, low-confidence sensing result on a high-value channel may justify a conservative modulation rate rather than a full channel avoidance.