Reinforcement Learning (RL)

Unlike traditional Electronic Protection Measures (EPM) that rely on pre-programmed rules, an RL agent learns directly from raw spectrum interaction. The agent observes the current spectrum occupancy and Signal-to-Interference-plus-Noise Ratio (SINR), takes an action such as switching frequency or adjusting power, and receives a reward based on Bit Error Rate (BER) or throughput. This trial-and-error loop allows the radio to adapt to novel smart jamming strategies—such as reactive jamming or bandwidth-hopping attacks—that were not anticipated during system design. The agent builds a policy that maps spectrum states to optimal transmission parameters without a mathematical model of the jammer's behavior.

The anti-jamming problem is formally structured as a Markov Decision Process (MDP) defined by the tuple (S, A, P, R). The state space S includes the current jamming-to-signal ratio (JSR) on active channels, historical spectrum waterfall data, and the radio's current transmission mode. The action space A consists of available countermeasures: selecting a new frequency channel, modifying the frequency hop spreading (FHSS) pattern, or adjusting the modulation and coding scheme. The transition probability P captures the stochastic nature of the jammer's next move, while the reward function R provides a positive scalar for successful packet delivery and a negative penalty for collisions or detected interference.

When the action space is discrete—such as selecting one of N available frequency channels—a Deep Q-Network (DQN) approximates the optimal action-value function Q*(s,a). The network takes a stack of recent spectrogram images or IQ sample statistics as input and outputs the expected cumulative reward for each possible channel choice. To stabilize training in the non-stationary jamming environment, the agent employs experience replay, storing transitions of (state, action, reward, next state) in a buffer and sampling random mini-batches to break temporal correlations. A separate target network with frozen weights is updated periodically to prevent harmful feedback loops during Q-value regression.

For continuous action spaces—such as tuning transmit power within a dBm range or steering a spatial filtering null with an adaptive antenna array—Policy Gradient methods like Proximal Policy Optimization (PPO) are preferred. Instead of learning value functions, the agent directly parameterizes a stochastic policy πθ(a|s) that outputs a probability distribution over actions. The policy is updated by estimating the gradient of expected reward and constraining the update step size to prevent catastrophic policy collapse. This is critical for proactive anti-jamming, where the radio must smoothly interpolate between aggressive power increases and subtle waveform modifications to maintain a low probability of intercept (LPI) profile.

The interaction between a cognitive radio and a smart jammer is inherently a non-cooperative game. As the RL agent improves its anti-jamming policy, an intelligent jammer using its own learning algorithm will adapt its strategy to defeat the new countermeasure. This creates a non-stationary environment that violates standard RL assumptions. Robust anti-jamming agents are trained using adversarial training or self-play, where the jammer's policy is simulated as a second learning agent. The radio's policy must converge to a Nash equilibrium or a robust strategy that performs well against a distribution of worst-case follower jamming and deceptive jamming attacks.

Training an RL agent directly on live hardware in a contested electromagnetic environment is impractical and risks damaging equipment. Instead, agents are trained in high-fidelity RF simulation environments that model multipath fading, Doppler shift, and realistic jammer waveforms including DRFM-based deceptive jamming. The trained policy is then transferred to a Software-Defined Radio (SDR) platform. To bridge the sim-to-real gap, domain randomization is applied during training—varying noise floors, channel impulse responses, and jammer parameters—forcing the policy to learn invariant features that generalize to the physical hardware's non-linearities and real-world propagation effects.

The mathematical framework underlying RL. An MDP formally defines the environment as a tuple (S, A, P, R, γ). S is the set of spectrum states (e.g., jammed, idle), A is the set of actions (e.g., switch frequency, increase power), P is the state transition probability, R is the reward signal (e.g., +1 for successful packet, -10 for collision), and γ is the discount factor. The agent's goal is to find a policy π(s) that maximizes the expected cumulative discounted reward.

A model-free, value-based RL algorithm. The agent learns an action-value function Q(s, a) that estimates the total reward expected by taking action a in state s and following the optimal policy thereafter. It is an off-policy algorithm, meaning it learns the optimal policy independently of the agent's current exploratory actions. In anti-jamming, Q-learning can learn to switch to the best channel without needing a model of the jammer's strategy.

An extension of Q-learning that uses a deep neural network to approximate the Q-value function, enabling RL in high-dimensional state spaces. Key innovations include:

• Experience Replay: Storing past transitions (s, a, r, s') in a buffer and sampling random mini-batches to break temporal correlations.
• Target Network: Using a separate, periodically updated network to compute target Q-values, stabilizing training. DQNs are widely used for learning anti-jamming policies directly from raw spectrum waterfall spectrograms.

A class of algorithms that directly parameterize and optimize the policy π(a|s) without requiring a value function. Unlike value-based methods, policy gradients can learn stochastic policies and naturally handle continuous action spaces (e.g., tuning transmission power). The REINFORCE algorithm and Actor-Critic architectures are common examples. These are ideal for generating smooth, adaptive jamming mitigation responses.

The fundamental dilemma in RL. Exploitation means choosing the best-known action (e.g., staying on a historically clean channel) to maximize immediate reward. Exploration means trying a suboptimal action (e.g., probing a new frequency) to discover potentially better long-term strategies. Common strategies include:

• ε-greedy: Choose a random action with probability ε.
• Upper Confidence Bound (UCB): Select actions based on their potential upside.
• Thompson Sampling: Probabilistically match the probability of an action being optimal.

The practice of engineering the reward function to accelerate learning by providing intermediate, dense feedback rather than sparse terminal rewards. In anti-jamming, a sparse reward might be +1 only for a successful file transfer. Reward shaping adds intermediate signals, such as:

• +0.1 for maintaining a link with a high SINR.
• -0.5 for detecting a follower jammer on the current channel.
• -0.01 per time step to penalize latency. Careful shaping prevents the agent from learning unintended behaviors.