Reward Function Design

The fundamental design choice between providing frequent feedback or only terminal signals.

• Sparse Rewards: Agent receives a signal only upon task completion (e.g., successful handover). Simpler to define but creates a challenging credit assignment problem where the agent struggles to identify which actions led to success.
• Dense Rewards: Agent receives incremental feedback at each timestep (e.g., small positive reward for moving toward target load balance). Accelerates learning but risks reward hacking where the agent exploits shortcuts.
• Practical approach: Combine both using potential-based reward shaping—a dense shaping reward that is mathematically guaranteed not to alter the optimal policy.

The technique of combining multiple competing objectives into a single scalar reward signal.

• Weighted linear sum: The most common approach: R = w₁·throughput - w₂·handover_failures - w₃·energy_consumption
• Weight tuning is critical: Small changes in relative weights can produce drastically different learned behaviors. Use Pareto frontier analysis to understand trade-offs before committing to weights.
• Dynamic weighting: Adjust weights based on network state—for example, prioritizing energy efficiency during low-load periods and throughput during peak hours.
• Constraint-based formulation: Frame hard requirements (e.g., maximum latency) as constraints rather than penalty terms, using constrained MDP or Lagrangian methods.

Domain-specific techniques for crafting rewards that accelerate learning in radio access network optimization.

• Throughput reward: R_throughput = Σᵢ log(1 + SINRᵢ) captures proportional fairness—diminishing returns prevent the agent from starving edge users to boost cell-center throughput.
• Handover penalty: Apply a fixed negative reward per handover event to discourage ping-pong handovers. Scale penalty by service type—VoLTE calls penalized more heavily than best-effort data.
• Energy reward: R_energy = -P_total where P_total includes power amplifier consumption and baseband processing. Use sleep mode transitions as discrete actions with associated wake-up latency penalties.
• QoE alignment: Map predicted Mean Opinion Score (MOS) to reward, ensuring the agent optimizes for perceived user experience rather than raw network KPIs.

Strategies for addressing the temporal gap between actions and their outcomes in network control.

• Temporal credit assignment: A load balancing action taken now may only show benefits seconds later when traffic patterns shift. Use eligibility traces (TD(λ)) to propagate credit backward through time.
• Discounted returns: G_t = Σₖ γᵏ·r_{t+k} where γ (discount factor) controls the agent's planning horizon. For RAN control loops operating on 10ms–1s timescales, γ values of 0.95–0.99 are typical.
• Terminal state definition: In continuing tasks (networks never stop), define pseudo-terminal states at episode boundaries (e.g., 1-hour intervals) or use average reward formulation instead of discounted returns.
• Hindsight experience replay: After an episode, relabel outcomes as if the agent had intended the achieved result, dramatically improving sample efficiency in sparse-reward environments.

Numerical techniques to stabilize training by controlling the scale and distribution of reward signals.

• Reward clipping: Constrain rewards to a fixed range (e.g., [-1, 1]) to prevent large gradients from destabilizing neural network training. Common in DQN and PPO implementations.
• Running normalization: Maintain an exponential moving average and standard deviation of rewards, standardizing each reward to zero mean and unit variance. Adapts to changing reward scales during learning.
• Advantage normalization: In actor-critic methods, normalize the advantage estimates within each batch rather than raw rewards, preserving relative action quality while controlling gradient magnitude.
• Caution: Aggressive clipping can destroy the ordinal relationship between rewards—the agent may lose the ability to distinguish between a moderately bad action and a catastrophic one.

An alternative paradigm where the reward function is inferred from demonstrations rather than hand-crafted.

• Core concept: Given trajectories from an expert policy (e.g., a heuristic load balancer or human operator), learn a reward function that explains the observed behavior.
• Maximum entropy IRL: Finds the reward function that makes the expert trajectories maximally probable while remaining as uncommitted as possible to unseen states—avoids overfitting to limited demonstrations.
• Generative Adversarial Imitation Learning (GAIL) : Uses a discriminator network to distinguish expert from agent trajectories, with the discriminator's output serving as a learned reward signal.
• RAN application: Extract reward functions from historical network optimization logs where experienced engineers manually tuned parameters, then deploy the learned reward for autonomous control at scale.

The process of mathematically balancing competing goals within a single reward function. In RAN, an agent must simultaneously maximize throughput while minimizing handover failures and energy consumption. This is typically achieved through a weighted sum: R = w1*Throughput - w2*HandoverFailures - w3*Energy. The challenge lies in tuning these weights to reflect true operational priorities without causing reward hacking, where the agent exploits a loophole in the formula to maximize reward without achieving the intended outcome.

A fundamental design choice dictating how frequently the agent receives feedback. A sparse reward is given only at the end of an episode (e.g., a successful handover), making credit assignment difficult but the objective simple. A dense reward provides continuous feedback (e.g., a small positive reward for every ms of latency reduction), which speeds up learning but requires careful potential-based shaping to prevent the agent from learning suboptimal, looping behaviors that maximize incremental gains rather than the final goal.

Framing the problem not as maximizing reward, but as satisfying a set of hard constraints. In Constrained Markov Decision Processes (CMDPs), the agent must maximize throughput while strictly ensuring that the probability of a handover failure remains below a threshold (e.g., < 0.1%). This is often implemented using a Lagrangian relaxation, where constraint violations incur a steep, dynamically adjusted penalty cost, ensuring the learned policy respects critical Service Level Agreements (SLAs).

A technique used when manually specifying a reward function is too complex. Instead of defining the reward, the agent observes expert demonstrations (e.g., a heuristic load-balancing algorithm) and infers the underlying reward function that the expert is implicitly optimizing. This is critical for capturing the unspoken trade-offs made by experienced network engineers, such as prioritizing voice traffic stability over peak data throughput during specific hours.

A method to accelerate learning by adding a potential-based reward F(s, s') = γΦ(s') - Φ(s) to the true environmental reward. This does not change the optimal policy but provides intermediate guidance. In predictive load balancing, Φ(s) could be a heuristic estimate of cell congestion. The agent gets a bonus for moving from a high-congestion state to a low-congestion state, injecting domain knowledge without distorting the final objective.

The mathematical objective the agent seeks to maximize: G_t = R_{t+1} + γR_{t+2} + γ^2R_{t+3} + .... The discount factor γ (gamma) determines the agent's farsightedness. A γ close to 0 makes the agent myopic, focusing on immediate throughput gains. A γ close to 1 makes it prioritize long-term network stability, accepting short-term congestion to prevent a future outage. Tuning gamma is essential for matching the operational planning horizon.