Reward Function

The granularity of the reward signal critically impacts learning speed and final policy quality.

• Dense Reward: Provides a scalar feedback at every timestep (e.g., a small penalty for every Watt of power consumed). This reduces the credit assignment problem but can lead to myopic, suboptimal local policies if not shaped carefully.
• Sparse Reward: Only signals success at the terminal state (e.g., +1 for a successful handover, 0 otherwise). This avoids injecting human bias but suffers from the exploration problem, where the agent never stumbles upon a reward in a vast state space.
• Practice: For RAN optimization, a hybrid approach is common: a dense auxiliary reward for maintaining a minimum SINR combined with a sparse, large terminal bonus for completing a user session without a radio link failure.

Reward shaping adds a heuristic bonus to the environment's native reward to accelerate learning, but naive shaping can change the optimal policy.

• Potential-Based Shaping: The only class of shaping functions guaranteed to preserve the original Markov Decision Process (MDP) optimal policy. It takes the form F(s, s') = γΦ(s') - Φ(s), where Φ is a state potential function.
• Application: In Dynamic Spectrum Access, a potential function Φ(s) could be the negative of the current interference level. The shaped reward encourages transitions to lower-interference states without altering the fundamental objective of maximizing throughput.
• Risk: Hand-crafted shaping functions that are not potential-based (e.g., arbitrary bonuses for "being near a target") can inadvertently teach the agent to oscillate or exploit the bonus rather than solve the task.

Real-world RAN problems involve conflicting objectives: maximize throughput, minimize latency, and minimize energy. These must be collapsed into a single scalar reward.

• Linear Scalarization: The most common method: R = w₁·R_throughput + w₂·R_latency + w₃·R_energy. The weights w define the trade-off, but finding the correct Pareto-optimal weighting is often a tedious hyperparameter search.
• Constraint-Based Formulation: A Lagrangian approach treats one objective as the primary reward and the others as constraints (e.g., "maximize throughput subject to latency < 10ms"). This is often implemented via a Lagrangian Multiplier that dynamically adjusts the penalty weight during training.
• Example: In Network Slicing, an eMBB slice might weight throughput at 0.9 and latency at 0.1, while a URLLC slice inverts these weights, all within the same physical infrastructure.

The discount factor γ ∈ [0, 1] determines the agent's horizon: how much it values future rewards over immediate ones.

• γ close to 0: The agent is myopic, maximizing only the immediate next reward. This is suitable for fast-fading channel decisions where the Channel State Information (CSI) is stale within milliseconds.
• γ close to 1: The agent is far-sighted, valuing long-term cumulative return. This is essential for Energy Efficiency Optimization, where turning off a power amplifier now saves energy over hours, even if it causes a minor immediate throughput dip.
• Non-Stationarity: In Multi-Agent Reinforcement Learning (MARL) for interference management, the environment is non-stationary from any single agent's perspective. A reward function that is stationary in a single-agent setting may fail to converge as other agents' policies change.

The magnitude and distribution of the reward signal directly control the variance of the policy gradient, impacting training stability.

• Reward Clipping: Bounding rewards to a fixed range (e.g., [-1, 1]) prevents a single anomalous high-reward transition from catastrophically updating the policy. This is standard practice in Proximal Policy Optimization (PPO).
• Running Normalization: Maintaining a running mean and standard deviation of historical rewards and whitening the current reward (r_normalized = (r - μ) / σ) keeps the scale of the advantage function consistent throughout training.
• RAN Context: A raw throughput reward of 100 Mbps vs. a latency penalty of 0.001 seconds creates a severe scale mismatch. Without normalization, the optimizer will ignore the latency signal entirely, producing a policy that maximizes speed but violates Quality of Service (QoS).

Beyond maximizing positive outcomes, the reward function must explicitly encode hard constraints and safety boundaries through negative signals.

• Large Negative Terminal Rewards: Assigning a substantial penalty (e.g., -100) upon entering a failure state like a Radio Link Failure (RLF) or violating a maximum transmit power constraint. This teaches the agent to avoid these regions at all costs.
• Action Penalties: A small per-step penalty on specific actions, such as a handover execution cost, prevents the ping-pong effect where the agent switches cells continuously for marginal gain.
• Curriculum Learning: The penalty magnitude can be annealed. Initially, a mild penalty for interference allows exploration; as training progresses, the penalty coefficient increases to enforce strict Interference Management in the final policy.

A foundational reward shaping strategy that directly incentivizes the agent to push more bits per second per Hertz through the network.

• Reward Signal: R = sum(SE_k) for all users k, where SE_k = log2(1 + SINR_k).
• Behavior Induced: The agent learns aggressive power allocation and MCS selection to maximize throughput.
• Pitfall: Pure SE maximization often leads to resource starvation for cell-edge users with poor SINR, as the agent ignores fairness.
• Use Case: Best suited for lightly loaded networks or premium-tier slices where raw capacity is the sole objective.

A concave utility function that balances total throughput with user fairness, preventing the starvation of cell-edge devices.

• Reward Signal: R = sum(log(T_k)) where T_k is the long-term average throughput of user k.
• Behavior Induced: The agent prioritizes users with high instantaneous channel quality relative to their own historical average, achieving a multi-user diversity gain.
• Mechanism: The logarithmic function has diminishing returns; doubling a high-throughput user's rate yields less marginal reward than improving a low-throughput user.
• Deployment: This is the de facto standard reward for many DRL-based scheduling agents in ns-3 and O-RAN testbeds.

A multi-objective reward that subtracts a power consumption cost from the throughput gain, driving the agent toward green networking.

• Reward Signal: R = w_1 * sum(SE_k) - w_2 * P_total, where P_total is the total transmit power and w_1, w_2 are weighting coefficients.
• Behavior Induced: The agent learns to selectively deactivate power amplifiers or reduce MIMO layers during low-load periods while maintaining QoS.
• Key Design Choice: The ratio w_2 / w_1 defines the operator's trade-off preference. A high ratio triggers aggressive sleep modes.
• Advanced Variant: Replace P_total with a dynamic power model that includes static circuit power consumption for more realistic hardware modeling.

A sparse, threshold-based reward that only provides positive feedback when strict latency or throughput guarantees are met, aligning the agent with SLA compliance.

• Reward Signal: R = (1/N) * sum(I(QoS_k >= QoS_target)), where I is an indicator function returning 1 if the target is met and 0 otherwise.
• Behavior Induced: The agent focuses on ensuring the minimum viable performance for all users rather than maximizing peak rates.
• Sparsity Challenge: A pure indicator function provides no gradient when the target is missed. To mitigate this, engineers often use a hinge loss variant: R = -sum(max(0, QoS_target - QoS_k)).
• Use Case: Critical for URLLC slices in 5G where 1ms latency guarantees are non-negotiable.

A reward function designed to optimize mobility robustness by explicitly penalizing unnecessary handovers and radio link failures.

• Reward Signal: R = w_1 * SE - w_2 * N_HO - w_3 * N_RLF, where N_HO is the handover count and N_RLF is the number of radio link failures.
• Behavior Induced: The agent learns to apply hysteresis dynamically, avoiding the ping-pong effect where a UE is repeatedly handed back and forth between two cells.
• Critical Insight: Without this penalty, a pure throughput agent will trigger a handover at every minor SINR fluctuation, generating massive signaling overhead in the core network.
• Implementation: The penalty coefficients w_2 and w_3 are typically tuned via a utility function that maps to Mean Opinion Score (MOS).

A staged reward shaping approach where the agent first learns basic coverage before optimizing capacity, preventing convergence to degenerate local optima.

• Stage 1 Reward: R = -sum(max(0, RSRP_target - RSRP_k)) — a dense negative reward that forces the agent to eliminate coverage holes by adjusting antenna tilts.
• Stage 2 Reward: Once 95% coverage is achieved, the reward transitions to a combined SE and EE objective.
• Behavior Induced: Prevents the agent from turning off cells to save energy at the cost of creating coverage blackouts.
• Mechanism: This is a form of potential-based reward shaping that preserves the optimal policy while accelerating convergence in complex heterogeneous networks.