Proximal Policy Optimization (PPO)

The defining mechanism of PPO that prevents destructively large policy updates. Instead of using a complex trust-region constraint like TRPO, PPO clips the probability ratio between the new and old policy.

• If the advantage is positive (good action), the ratio is clipped at 1 + ε to prevent over-optimizing.
• If the advantage is negative (bad action), the ratio is clipped at 1 - ε to prevent over-penalizing.
• The final objective takes the minimum of the clipped and unclipped surrogate, creating a pessimistic bound.

This simple clipping mechanism keeps the new policy within a trust region of the old policy, ensuring stable learning without the computational overhead of second-order optimization.

PPO is an on-policy algorithm, meaning it learns from trajectories collected by the current policy, not from a replay buffer of past experiences. To improve sample efficiency, PPO uses importance sampling to reuse collected data for multiple gradient updates.

• The agent collects a batch of trajectories using the current policy π_θ_old.
• It then performs multiple epochs of mini-batch updates on this same data.
• The probability ratio r_t(θ) = π_θ(a_t|s_t) / π_θ_old(a_t|s_t) corrects for the distribution shift.
• Without clipping, this ratio could diverge; the clipped objective keeps the correction valid.

This approach bridges the gap between on-policy stability and off-policy data efficiency, critical for slow-changing RAN environments.

PPO typically uses Generalized Advantage Estimation (GAE) to compute the advantage function, which tells the agent how much better an action was than expected. GAE balances bias and variance through an exponential weighting parameter λ.

• λ = 0: High bias, low variance (1-step TD error).
• λ = 1: Low bias, high variance (Monte Carlo returns).
• Typical value: λ = 0.95 provides a smooth trade-off.

GAE computes a weighted sum of n-step temporal difference errors, providing a stable learning signal even with noisy wireless channel rewards. This is essential for RAN tasks like power control, where the impact of an action may be delayed by several transmission time intervals.

PPO employs a dual-network actor-critic architecture that separates policy learning from value estimation, reducing gradient variance compared to pure policy gradient methods.

• Actor (Policy Network): Outputs the mean and standard deviation of a Gaussian distribution over continuous actions (e.g., transmission power levels, beam angles). Updated using the clipped surrogate objective.
• Critic (Value Network): Estimates the state-value function V(s), trained to minimize the mean squared error against discounted returns.
• Shared or Separate Backbones: In RAN applications, the actor and critic often share initial convolutional or fully-connected layers to extract features from channel state information, then branch into separate heads.

This separation allows the critic to provide a stable baseline for the advantage calculation, accelerating convergence in non-stationary wireless environments.

To prevent premature convergence to suboptimal policies, PPO adds an entropy bonus to the loss function. This term encourages the policy distribution to remain stochastic and exploratory.

• The total loss: L = L_clip + c_1 * L_value - c_2 * H(π)
• H(π) is the entropy of the action distribution.
• Higher entropy means more random exploration; lower entropy means more deterministic exploitation.
• The coefficient c_2 is typically small (e.g., 0.01) and can be annealed over time.

In dynamic spectrum access, this entropy bonus prevents the agent from locking onto a single frequency channel and encourages it to continuously probe alternative bands that may have become available.

Unlike DQN, which is limited to discrete actions, PPO natively handles continuous action spaces by outputting the parameters of a probability distribution—typically a multivariate Gaussian with a diagonal covariance matrix.

• The actor outputs a mean vector μ and a log-standard-deviation vector log σ for each action dimension.
• Actions are sampled: a ~ N(μ, diag(σ²)).
• The log-probability of the sampled action is computed analytically for the policy gradient.
• For bounded actions (e.g., power between 0 and max), a tanh squashing function can be applied.

This makes PPO ideal for RAN control tasks like continuous power control, beamforming weight optimization, and adaptive modulation parameter tuning, where actions are real-valued vectors rather than discrete choices.