Reward Shaping

The gold standard of reward shaping that guarantees policy invariance—the optimal policy in the shaped environment remains optimal in the original MDP. It defines a potential function Φ(s) over states, and the shaping reward is the difference in potential: F(s, s') = γΦ(s') - Φ(s). This ensures the agent is guided toward high-potential states without distorting the true objective. In cognitive radio, potential functions often encode distance to known spectrum holes or signal-to-noise ratio gradients.

A practical shaping strategy that rewards the agent for reducing its distance to a goal state. In dynamic spectrum access, this translates to:

• Spectral proximity bonus: Reward for tuning closer to an identified vacant channel
• Interference minimization: Positive feedback when transmission power is adjusted away from primary user thresholds
• Handoff efficiency: Shaping rewards for minimizing the time spent searching during spectrum handoffs These heuristics are particularly effective when the goal state is well-defined but the path to reach it is combinatorially complex.

An intrinsic motivation approach where the agent receives bonus rewards for visiting novel or unpredictable states, combating the exploration-exploitation dilemma in sparse-reward spectrum environments. Implementations include:

• Intrinsic Curiosity Module (ICM): Predicts the next state given current state and action; rewards prediction error
• Random Network Distillation (RND): Uses a fixed random target network and a predictor network; novelty is measured by prediction error In jamming environments, curiosity bonuses drive agents to explore unconventional frequency bands that deterministic policies would ignore.

Breaking a complex spectrum access task into a hierarchical sequence of intermediate milestones, each with its own shaping reward. For cognitive radio anti-jamming:

• Detection subgoal: Reward for correctly classifying jammer type
• Evasion subgoal: Bonus for successfully predicting jammer sweep pattern
• Reconnection subgoal: Shaping reward for re-establishing link on clean channel This approach leverages temporal abstraction to provide dense feedback in what would otherwise be a single sparse terminal reward upon full link restoration.

When hand-crafted potential functions are infeasible, inverse reinforcement learning (IRL) infers a reward function from expert demonstrations of optimal spectrum access behavior. The recovered reward function serves as the shaping signal. Key methods:

• Maximum Entropy IRL: Assumes expert behavior maximizes entropy while matching feature expectations
• Generative Adversarial Imitation Learning (GAIL): Uses a discriminator to distinguish agent from expert trajectories This is particularly valuable in contested RF environments where human operators possess tacit knowledge difficult to encode explicitly.

Dynamic adjustment of shaping reward magnitude to prevent reward hacking—where agents exploit shaping signals at the expense of true task objectives. Techniques include:

• Annealed shaping: Gradually decay the shaping coefficient λ as training progresses, allowing the agent to transition from guided exploration to pure task optimization
• PopArt normalization: Adaptive normalization of value targets to handle varying reward scales across frequency bands
• Conditional shaping: Activate auxiliary rewards only when the agent's policy entropy drops below a threshold, targeting plateaus This ensures the agent internalizes the true spectrum access objective rather than becoming dependent on artificial guidance.

The fundamental challenge that reward shaping solves. In dynamic spectrum access, an agent may transmit for thousands of time steps before receiving a binary success/failure signal. Without intermediate guidance, the agent fails to associate specific actions with outcomes. Potential-based reward shaping injects domain knowledge—such as received signal strength indicator (RSSI) gradients or channel quality estimates—to provide incremental feedback, transforming a flat reward landscape into a navigable gradient for faster policy convergence.

The mathematically rigorous form of reward shaping that guarantees policy invariance. A shaping reward F(s,s') is defined as γΦ(s') - Φ(s), where Φ is a state potential function and γ is the discount factor. This formulation ensures the optimal policy in the shaped MDP remains identical to the original. For cognitive radio, Φ might encode the estimated channel capacity or distance to a jammer, providing dense feedback without distorting the agent's ultimate objective of maximizing throughput.

Integrating reward shaping into model-free reinforcement learning accelerates Q-table or deep Q-network (DQN) convergence. The agent receives r + F(s,s') at each transition rather than the sparse environmental reward alone. In spectrum access, a shaped reward might combine:

• Positive shaping: +0.1 for selecting a historically idle channel
• Negative shaping: -0.5 for approaching a known primary user guard band This guides exploration toward promising regions of the action space while the agent refines its value estimates.

Reward shaping directly influences the exploration-exploitation balance. Overly aggressive shaping can trap an agent in local optima by discouraging exploration of novel spectrum opportunities. Conversely, well-calibrated shaping reduces the exploration burden by encoding prior knowledge, allowing the agent to exploit high-reward channels sooner. Techniques like Boltzmann exploration with shaped Q-values or Thompson sampling guided by shaped posterior distributions help maintain adaptive behavior in non-stationary RF environments.

A critical application where reward shaping dramatically reduces training time. An agent learning to evade a reactive jammer receives sparse rewards only when communication succeeds. Shaping functions based on:

• Spectral occupancy patterns detected via FFT
• Jammer signal strength gradients
• Historical channel availability statistics provide immediate feedback on evasion maneuvers. This enables the cognitive radio to learn effective frequency-hopping patterns in minutes rather than hours of over-the-air interaction.

In actor-critic architectures like PPO, reward shaping reduces the variance of policy gradient estimates. The critic network learns a shaped value function V(s) + Φ(s), while the actor receives dense advantage signals. For cognitive radio control, this stabilizes training when learning continuous power control or beamforming parameters. The shaping potential Φ can be a learned function itself—a shaping network trained to predict long-term spectrum utility, creating an adaptive reward curriculum that evolves with the agent's proficiency.