Deep Q-Network (DQN)

A biologically inspired mechanism that decouples data generation from learning. Instead of learning from sequential, temporally correlated experiences, the agent stores transitions—(state, action, reward, next_state) tuples—in a replay buffer and samples random mini-batches during training.

• Breaks Temporal Correlation: Prevents the network from overfitting to recent, highly correlated sequences of observations.
• Improves Data Efficiency: Each experience can be reused multiple times for weight updates, reducing the number of interactions needed with the environment.
• Stabilizes Learning: By averaging the Q-value updates over a diverse set of past experiences, the training distribution becomes more stationary, satisfying the i.i.d. assumptions of stochastic gradient descent.

A secondary neural network with an identical architecture to the primary Q-network, used to compute the Temporal Difference (TD) target. Its parameters are held fixed for a set number of steps and only periodically updated by copying the weights of the primary network.

• Mitigates Moving Target Problem: Without a target network, the Q-values used to compute the TD target shift with every gradient update, causing harmful feedback loops and divergence.
• Stabilizes Bootstrapping: Provides a quasi-stationary objective for the primary network to regress toward during each training iteration.
• Update Mechanism: A hard update copies weights every C steps, while a soft update (Polyak averaging) blends them incrementally: θ_target ← τθ_primary + (1 − τ)θ_target.

DQN replaces the traditional tabular Q-table with a deep convolutional neural network (CNN) that maps raw pixel inputs directly to action-value estimates for each possible action. This allows the agent to operate in high-dimensional state spaces where enumerating every state is impossible.

• End-to-End Learning: The network learns to extract relevant visual features—edges, textures, objects—directly from raw frames without hand-crafted feature engineering.
• Single Forward Pass: The architecture outputs a Q-value for every discrete action simultaneously, enabling efficient action selection via a single inference step.
• Input Preprocessing: Frames are typically downsampled, converted to grayscale, and stacked across multiple timesteps to provide a sense of motion and temporal context.

A simple but effective exploration strategy where the agent selects a random action with probability ε and the greedy action (max Q-value) with probability 1 − ε. The value of ε is typically annealed from 1.0 to a small constant over the course of training.

• Ensures Coverage: Forces the agent to explore the state-action space sufficiently to discover high-reward trajectories before committing to a policy.
• Annealing Schedule: Linear or exponential decay from ε_start to ε_end over a specified number of frames balances early exploration with late-stage exploitation.
• Limitation: Undirected exploration is inefficient in environments with sparse rewards, motivating more sophisticated strategies like Boltzmann exploration or intrinsic curiosity.

To simplify the reward scale and enable a single learning rate to work across diverse games, all positive rewards are clipped to +1, all negative rewards to −1, and zero rewards remain 0.

• Scale Invariance: Prevents the network's gradients from exploding or vanishing due to vastly different reward magnitudes across environments.
• Limits Over-Optimism: Restrains the unbounded growth of Q-values that can occur when positive rewards accumulate without bound.
• Trade-off: Discards the magnitude information of the original reward, which can make it harder to distinguish between marginally good and exceptionally good outcomes.

A single static frame cannot convey velocity or direction. DQN addresses this by stacking the last 4 preprocessed frames into a single input tensor, providing the CNN with a short-term motion history.

• Partial Observability Mitigation: Transforms a Partially Observable Markov Decision Process (POMDP) into an effective MDP by including recent observations in the state representation.
• Motion Detection: Enables the network to infer speed, acceleration, and trajectory of objects—critical for games like Breakout or Pong.
• Implementation: The stacked frames form a tensor of shape (84, 84, 4) fed as input to the first convolutional layer.

A biologically-inspired mechanism that stores agent transitions (state, action, reward, next state) in a replay buffer and uniformly samples random mini-batches for training.

• Purpose: Breaks temporal correlations between consecutive samples that would otherwise destabilize neural network training.
• Mechanism: Transitions are stored in a circular buffer of fixed capacity (e.g., 1 million tuples). During each training step, a mini-batch of 32-64 transitions is randomly drawn.
• Efficiency: Each transition is reused multiple times, dramatically improving sample efficiency compared to online Q-learning.
• Limitation: Uniform sampling treats all transitions equally; Prioritized Experience Replay extends this by sampling high-TD-error transitions more frequently.

A separate neural network with identical architecture to the online Q-network but with frozen parameters that are periodically updated, used to compute stable TD targets.

• Instability Problem: Using the same network to both select and evaluate actions creates a moving target, causing harmful feedback loops and divergence.
• Solution: The target network's weights are held fixed for C steps (e.g., 10,000) and then copied from the online network, providing a stationary learning target.
• Loss Function: The TD error is computed as r + γ max_a' Q_target(s', a') - Q_online(s, a), where gradients flow only through the online network.
• Soft Updates: A variant uses Polyak averaging: θ_target ← τθ_online + (1-τ)θ_target with a small τ (e.g., 0.001) for smoother adaptation.

An extension that decouples action selection from action evaluation to address the overestimation bias inherent in standard DQN.

• Overestimation Problem: The max operator in the Bellman update uses the same values for both choosing and evaluating actions, systematically overestimating Q-values, especially in noisy environments.
• Decoupling: The online network selects the best action: a* = argmax_a Q_online(s', a). The target network then evaluates that action: Q_target(s', a*).
• Impact: Produces more accurate value estimates, leading to better policies, particularly in stochastic domains like wireless channel scheduling.
• Minimal Overhead: Requires no additional networks—just a change in the TD target computation—making it a standard default in modern implementations.

A network architecture that splits the Q-value estimation into two separate streams: the state value function V(s) and the advantage function A(s, a).

• Architecture: A shared convolutional or fully-connected backbone feeds into two separate heads. The streams are recombined as Q(s,a) = V(s) + A(s,a) - mean(A(s,a)).
• Key Insight: For many states, the exact action choice matters little (e.g., a user in good channel conditions). The dueling architecture learns to identify these situations without needing to evaluate every action's marginal benefit.
• RAN Application: In resource block allocation, the value stream can quickly assess overall cell load while the advantage stream focuses on per-user scheduling gains.
• Training Efficiency: Learns the state value more rapidly because it's updated with every experience, not just the chosen action's experience.

An improvement over uniform sampling that assigns higher replay probability to transitions with larger temporal-difference (TD) errors, focusing learning on surprising or high-information events.

• Priority Metric: p_i = |δ_i| + ε, where δ is the TD error and ε is a small constant ensuring non-zero probability for all transitions.
• Sampling Probability: P(i) = p_i^α / Σ_k p_k^α, where α controls the degree of prioritization (α=0 recovers uniform sampling).
• Importance Sampling: Weights are applied to correct the bias introduced by non-uniform sampling: w_i = (N · P(i))^{-β}, with β annealed to 1 over training.
• RAN Relevance: Rare but critical events like sudden interference spikes or handover failures receive more frequent replays, accelerating learning of robust policies.

The recursive decomposition that defines the optimal action-value function as the immediate reward plus the discounted maximum future value, forming the theoretical backbone of all Q-learning variants.

• Optimality Equation: Q*(s,a) = E[r + γ max_a' Q*(s', a') | s, a], where γ ∈ [0,1] is the discount factor.
• Temporal Difference: DQN minimizes the difference between the left and right sides of this equation, treating the right side as a fixed target computed via the target network.
• Discount Factor: γ controls the agent's horizon—γ close to 1 emphasizes long-term rewards (e.g., sustained spectral efficiency), while γ close to 0 prioritizes immediate gains.
• Convergence: Under the tabular setting with infinite visits, Q-learning provably converges to Q*. Function approximation with neural networks provides no such guarantee, motivating the stabilization techniques above.