Deep Reinforcement Learning (DRL)

Unlike static pattern recognition, DRL operates on temporal sequences where actions influence future states. The agent's goal is to maximize the cumulative discounted reward over a trajectory, not just immediate gain.

• Markov Property: Decisions depend only on the current state, not the full history.
• Credit Assignment: The agent must learn which actions in a long sequence were responsible for a delayed reward.
• Example: A base station scheduler deciding which user to serve in each 1ms Transmission Time Interval (TTI) to maximize long-term throughput.

DRL replaces traditional lookup tables with deep neural networks to handle high-dimensional, continuous state spaces that are impossible to discretize. The network generalizes across similar states.

• Value Networks: Approximate the expected return Q(s,a) or V(s) for state-action pairs.
• Policy Networks: Directly output a probability distribution over actions π(a|s) without computing values.
• Example: A convolutional neural network processing a grid of SINR measurements across hundreds of cells to output power control decisions.

The agent must balance exploiting known high-reward actions against exploring untried actions that might yield even higher returns. This is the central dilemma of reinforcement learning.

• ε-greedy: Selects a random action with probability ε, otherwise exploits.
• Entropy Regularization: Adds a bonus to the objective function for maintaining a stochastic, high-entropy policy (used in Soft Actor-Critic).
• Example: A handover optimization agent occasionally testing a non-optimal target cell to discover if interference patterns have shifted.

DRL algorithms are categorized by whether they learn an explicit world model of environment dynamics.

• Model-Free: Learns policy or value directly from sampled transitions without predicting next states. Dominant in complex RAN tasks due to simulation fidelity challenges.
• Model-Based: Learns a predictive model P(s'|s,a) and uses it for planning. Enables sample efficiency but suffers from compounding model errors.
• Example: Model-free PPO directly learning a scheduling policy from ns-3 simulator interactions vs. model-based planning with a learned digital twin of the network.

This distinction defines whether the agent learns from data generated by its current policy or from a different behavioral policy.

• On-Policy (e.g., PPO): Requires fresh data from the latest policy; discards old trajectories. More stable but less sample-efficient.
• Off-Policy (e.g., DQN, SAC): Can reuse historical data stored in a replay buffer. Enables learning from demonstrations or offline datasets.
• Example: An off-policy SAC agent learning optimal beamforming by reusing millions of past channel realizations stored in a replay buffer.

The reward function is the sole mechanism for communicating task objectives to the agent. Poorly designed rewards lead to reward hacking, where the agent achieves high scores through unintended behaviors.

• Sparse Rewards: Only given at terminal states (e.g., connection drop). Hard to learn from.
• Shaped Rewards: Dense, incremental signals (e.g., SINR improvement) that guide learning but risk biasing the policy.
• Multi-Objective Rewards: Weighted sum of competing goals like throughput maximization and energy minimization.
• Example: A load balancing agent receiving a reward of +1 for each user above the QoS threshold and -10 for each dropped call.

The formal mathematical framework underlying all DRL-based RAN control. An MDP models the cellular environment as a tuple (S, A, P, R, γ) where:

• S: State space (e.g., buffer statuses, SINR distributions, user positions)
• A: Action space (e.g., power adjustments, RB allocations, MCS selection)
• P: Transition probabilities capturing channel fading and user mobility
• R: Reward function encoding QoS targets and energy penalties
• γ: Discount factor balancing immediate vs. future network performance

The Markov property—that future states depend only on the current state and action—is a critical assumption when framing RRM as a sequential decision problem.

A foundational value-based DRL algorithm that approximates the optimal action-value function Q(s,a)* using a deep neural network. For RAN applications, DQN introduced two critical innovations:

• Experience Replay: Stores past (state, action, reward, next state) transitions from the live network in a buffer, sampling random mini-batches to break harmful temporal correlations in cellular traffic data
• Target Network: Maintains a separate, periodically updated copy of the Q-network to stabilize training against the non-stationary target values common in dynamic wireless environments

DQN is naturally suited for discrete action spaces, such as selecting from a finite set of modulation and coding schemes or choosing which user to schedule in a given TTI.

A hybrid DRL paradigm that combines the strengths of policy-based and value-based methods, widely adopted for continuous RAN control tasks. The architecture consists of two neural networks:

• Actor (Policy Network): Directly outputs the action to take—such as a continuous transmit power level or beamforming weight vector—parameterizing the policy π(a|s)
• Critic (Value Network): Estimates the value function V(s) or Q(s,a) to evaluate the actor's decisions, reducing the high variance of pure policy gradient methods

This architecture is the backbone of algorithms like A3C, PPO, and SAC, enabling fine-grained control over continuous parameters like power allocation and tilt angle optimization in massive MIMO systems.

An on-policy policy gradient algorithm that has become the de facto standard for many RAN optimization tasks due to its stability and sample efficiency. PPO constrains policy updates using a clipped surrogate objective function:

• Prevents destructively large policy updates that could degrade live network performance during online learning
• Operates within a trust region, ensuring monotonic improvement in expected cumulative reward
• Balances exploration of new scheduling strategies with exploitation of known good configurations

PPO is particularly effective for network slicing orchestration and dynamic spectrum access, where unstable policies could cause service-level agreement violations or harmful interference to primary users.

An off-policy maximum entropy DRL algorithm that excels in continuous action spaces and exploration-heavy RAN environments. SAC augments the standard reward objective with an entropy bonus that encourages the policy to remain stochastic:

• Prevents premature convergence to suboptimal power control or beamforming strategies
• Automatically tunes the temperature parameter α to balance exploration and exploitation
• Off-policy design enables high sample efficiency by reusing past network telemetry data

SAC is particularly well-suited for joint power and resource block allocation in multi-cell OFDMA systems, where the action space is high-dimensional and continuous, and robust exploration prevents the agent from settling into local optima caused by complex interference patterns.

An extension of DRL where multiple autonomous agents—typically one per base station or network slice—interact within the shared electromagnetic environment. Key paradigms include:

• Cooperative MARL: Agents collaborate to maximize a global objective, such as sum-rate or energy efficiency across the entire RAN
• Competitive MARL: Agents compete for scarce spectrum resources, modeled as a stochastic game
• Mixed MARL: Combines cooperation and competition, relevant for heterogeneous networks with both macro and small cells

The dominant training paradigm is Centralized Training Decentralized Execution (CTDE), where agents access global state information during offline training but act using only local observations during deployment, eliminating inter-base-station communication overhead.