Deep Q-Network (DQN)

A biological-inspired mechanism that breaks the correlation between consecutive training samples. Instead of learning from sequential transitions, the agent stores its experiences—defined as state, action, reward, next state tuples—in a replay memory buffer. During training, it randomly samples mini-batches from this buffer. This technique satisfies the i.i.d. assumption required by stochastic gradient descent and allows the network to reuse rare, high-value experiences multiple times, dramatically improving data efficiency and learning stability.

A critical stabilization trick that decouples the network being trained from the network used to generate the target values. DQN maintains two identical neural networks: a policy network and a target network. The target network's weights are frozen and only periodically updated (e.g., every 10,000 steps) by copying the policy network's parameters. This prevents the harmful feedback loop where updates to the network immediately shift the target it is trying to reach, mitigating oscillations and divergence.

A simple but effective strategy to balance the exploration-exploitation dilemma. The agent selects a random action with probability epsilon (ε) and the greedy action (the one with the highest predicted Q-value) with probability 1-ε. Epsilon is typically annealed from a high value (e.g., 1.0) to a low final value (e.g., 0.01) over millions of steps. This ensures the agent extensively explores the state-action space early in training before gradually shifting its focus to exploiting the learned policy.

The original DQN demonstrated that a deep neural network could directly map raw, high-dimensional sensory input to action values. It used a convolutional neural network (CNN) to process raw pixel frames from Atari 2600 games, learning hierarchical visual features without manual feature engineering. The architecture typically consists of:

• 3 convolutional layers for spatial feature extraction
• Fully connected layers to output a Q-value for each possible action This end-to-end learning from pixels to controls was a landmark achievement.

The mathematical framework that formalizes the caching problem DQN solves. An MDP defines the environment using:

• States: Current cache contents, predicted content popularity, and network load
• Actions: Which content to evict and which to pre-fetch
• Transition Model: Probability of the next state given the current state and action
• Reward: A cache utility function that penalizes cache misses and rewards hits

DQN learns an optimal policy by maximizing cumulative discounted reward without requiring an explicit transition model.

A critical DQN innovation that stabilizes training by breaking temporal correlations in sequential data. The agent stores transitions (state, action, reward, next state) in a replay buffer and samples random mini-batches for training.

In caching contexts, this prevents the network from overfitting to recent request patterns and enables it to learn from rare but important events like sudden content viral spikes. Prioritized variants weight transitions by temporal-difference error to focus learning on surprising outcomes.

A separate, periodically updated copy of the Q-network that addresses the moving target problem in reinforcement learning. Without it, the network chases its own predictions, causing harmful feedback loops.

The target network's weights are frozen for a fixed number of steps before being updated via a soft update (Polyak averaging) or hard replacement. This decoupling is essential for learning stable cache replacement policies in non-stationary content popularity environments.

A simpler alternative to full DQN for caching when the state space is limited. Bandit algorithms like Thompson Sampling and Upper Confidence Bound (UCB) solve the exploration-exploitation dilemma without modeling state transitions.

• Exploration: Trying new content to discover emerging popularity
• Exploitation: Caching known popular content to maximize immediate hit rate

Bandits are computationally lighter than DQN but cannot capture sequential dependencies or long-term consequences of eviction decisions.

An enhancement that addresses DQN's tendency to overestimate Q-values by decoupling action selection from evaluation. The online network selects the best action, but the target network evaluates it.

This prevents the pathological feedback where overestimated values for suboptimal caching actions propagate through bootstrapping. In proactive caching, Double DQN produces more conservative and accurate value estimates, leading to better eviction decisions under uncertainty.

A neural architecture that splits the Q-network into two streams:

• Value stream: Estimates the inherent value of being in a cache state
• Advantage stream: Estimates the relative benefit of each caching action

This decomposition is particularly useful when many actions have similar values—common in large content catalogs where most items have near-zero popularity. The dueling structure learns which states are valuable regardless of the specific action taken.