Reinforcement Learning Link Adaptation

The link adaptation problem is formalized as a Markov Decision Process (MDP). The state is defined by the Channel Quality Indicator (CQI), past ACK/NACK history, and HARQ retransmission count. The action is the selection of a Modulation and Coding Scheme (MCS) index. The reward function balances spectral efficiency against a penalty for Block Error Rate (BLER) exceeding a target threshold, typically 10%.

• State Space: CQI, SINR, Doppler spread, HARQ status
• Action Space: Discrete MCS indices (e.g., QPSK to 256-QAM)
• Reward: Throughput minus a weighted BLER violation penalty

A Deep Q-Network approximates the optimal action-value function Q(s,a), predicting the long-term cumulative reward for selecting a specific MCS in a given channel state. The agent uses an epsilon-greedy exploration strategy to occasionally try non-greedy MCS choices, gathering data on their outcomes. An experience replay buffer stores transitions (state, action, reward, next state) to break temporal correlations and stabilize training.

• Input: Current CQI and HARQ feedback vector
• Output: Q-values for each available MCS index
• Training: Minimize Bellman error using a target network updated periodically

Traditional Outer Loop Link Adaptation (OLLA) applies a static offset to SINR estimates based on long-term BLER averages. It is reactive and slow to adapt. An RL agent learns a non-linear, context-aware mapping from channel state directly to MCS, capturing complex temporal dependencies that a simple offset cannot.

• OLLA: Single scalar SINR offset, slow convergence
• RL Agent: Learns joint distribution of CQI, Doppler, and interference patterns
• Advantage: RL anticipates fast fading dips and preemptively lowers MCS, avoiding burst errors that OLLA only corrects after the fact

While DQN handles discrete MCS selection, Policy Gradient methods like Proximal Policy Optimization (PPO) can learn a stochastic policy directly. This is useful when the optimal strategy involves probabilistic MCS selection to probe channel capacity. The policy network outputs a probability distribution over MCS indices, and the agent samples from it.

• Actor Network: Outputs mean and variance of a Gaussian policy over MCS
• Critic Network: Estimates the value function for variance reduction
• Benefit: Naturally handles the exploration-exploitation trade-off without a separate epsilon schedule

In real networks, ACK/NACK feedback arrives with a delay of several Transmission Time Intervals (TTIs). This turns the problem into a Partially Observable Markov Decision Process (POMDP). The RL agent must maintain an internal belief state over the true channel condition. Architectures like Deep Recurrent Q-Networks (DRQN) use LSTM layers to integrate observations over time, implicitly learning to handle feedback latency.

• Challenge: Action taken at time t receives feedback at t+k
• Solution: Stack past CQI and HARQ observations into a history window
• Architecture: LSTM or Transformer encoder before the Q-value head

Training an RL agent directly on a live network is risky and slow. Instead, agents are trained in a channel emulator that models 3GPP CDL/TDL channel profiles with varying delay spread, Doppler, and SNR. Domain randomization varies these parameters beyond expected operational ranges during training, forcing the agent to learn robust policies that transfer to real hardware without fine-tuning.

• Training Environment: GPU-accelerated channel simulator with 3GPP models
• Randomization: SNR range, Doppler shift, interference burst patterns
• Result: Zero-shot deployment on software-defined radio (SDR) testbeds

Also known as deep unfolding, this methodology unrolls the iterations of a classical optimization algorithm into a neural network where each layer corresponds to one iteration. For link adaptation, this bridges the gap between purely data-driven RL and established communication theory.

• Example: Learned ISTA for sparse recovery.
• Advantage: Combines the interpretability of model-based methods with the adaptability of deep learning.
• Application: Can be used to design the reward function or constrain the action space of the RL agent to ensure physically realizable MCS selections.

In millimeter-wave systems, link adaptation is tightly coupled with beam management. RL agents can jointly optimize beam direction and MCS selection to combat intermittent blockages.

• Sensor Fusion: Uses sub-6GHz channel information, LiDAR, or GPS as auxiliary inputs.
• Latency Reduction: Eliminates the need for exhaustive beam sweeping before data transmission.
• Joint Optimization: The RL state space expands to include beam indices, enabling holistic link configuration.

A hybrid model-based deep learning architecture that integrates the structural flow of the Kalman filter with small neural networks. For link adaptation, KalmanNet provides a robust mechanism for tracking channel dynamics over time.

• Function: Learns unknown system dynamics and noise statistics directly from data.
• Role in RL: Can serve as the state estimator that feeds the RL agent a denoised, temporally coherent view of the channel.
• Outcome: Stabilizes MCS switching, preventing the agent from oscillating between rates due to noisy instantaneous feedback.

A paradigm that treats the entire physical layer as a single autoencoder neural network. This contrasts with modular RL link adaptation by jointly optimizing the transmitter, channel, and receiver for a specific task.

• Contrast: RL link adaptation typically optimizes MCS selection within a standard-compliant PHY, while end-to-end learning redesigns the waveforms themselves.
• Trade-off: End-to-end systems can outperform classical architectures on specific metrics but sacrifice interoperability with existing standards like 5G NR.
• Convergence: Future systems may use RL for higher-level configuration of an end-to-end learned PHY core.

The non-linear distortion introduced by power amplifiers (PAs) fundamentally alters the effective signal-to-noise ratio, making the MCS table's theoretical thresholds inaccurate. Neural network DPD learns to invert this non-linear transfer function.

• Impact on RL: Linearizing the PA output ensures that the RL agent's chosen MCS actually experiences the expected channel conditions.
• Joint Learning: Advanced architectures co-train the DPD network and the link adaptation policy to maximize end-to-end throughput under a shared power efficiency constraint.
• Hardware: Deployed on FPGAs near the radio front-end for real-time correction.