Deep Q-Network (DQN)

A biologically inspired memory mechanism that decouples data generation from learning. Instead of learning from sequential experiences as they occur, the agent stores each transition—(state, action, reward, next state)—in a replay buffer and uniformly samples random mini-batches for training.

This breaks the harmful temporal correlations between consecutive samples that would otherwise destabilize neural network training. Each experience is reused multiple times, dramatically improving sample efficiency.

• Buffer capacity: Typically 1 million transitions
• Mini-batch size: Commonly 32 or 64
• Sampling strategy: Uniform random (later variants use prioritized sampling)

A separate neural network with identical architecture to the primary Q-network but with frozen parameters that are updated only periodically. This addresses the moving target problem where the network chases its own bootstrapped estimates.

The primary network computes Q-values for action selection, while the target network computes the stable Q-value targets for the loss function. The target network's weights are updated every C steps by copying the primary network's weights.

• Update frequency: Every 10,000 steps in original DQN
• Loss function: Mean squared error between predicted Q and target Q
• Stabilization: Prevents harmful feedback loops and divergence

The original DQN used a deep convolutional neural network to process raw pixel inputs directly, eliminating the need for hand-crafted feature engineering. The architecture learned hierarchical visual representations end-to-end.

The network processed 84×84×4 input frames (4 grayscale frames stacked for temporal context) through three convolutional layers followed by two fully connected layers, outputting Q-values for each possible action.

• Input: 84×84×4 preprocessed frames
• Architecture: 3 conv layers + 2 FC layers
• Output: Q-value per valid action (4-18 actions depending on game)

All positive rewards are clipped to +1, all negative rewards to -1, and zero remains zero. This simple transformation ensures the error gradients remain bounded across vastly different games with incompatible reward scales.

Without clipping, a game with rewards in the thousands would produce gradients orders of magnitude larger than a game with rewards in single digits, making a single set of hyperparameters impossible. Clipping trades fine-grained reward distinction for universal training stability.

• Positive rewards: Clamped to +1
• Negative rewards: Clamped to -1
• Trade-off: Loses reward magnitude information but enables single architecture across domains

The agent selects an action every k frames and repeats that action for the intervening frames. In the original DQN, k=4, meaning the agent operates at roughly 15 Hz on 60 FPS games.

This technique reduces computational load without meaningful loss of control granularity, as most Atari games don't require frame-level decisions. It also increases the temporal distance between consecutive states in the replay buffer, further decorrelating training samples.

• Action repeat: 4 frames per decision
• Effective frame rate: 15 FPS on 60 FPS environments
• Benefit: 4x reduction in forward passes during training

The exploration rate ε starts at 1.0 (pure random actions) and is annealed linearly to a final value of 0.1 over the first million frames, then held constant. This ensures the agent extensively explores the state space early in training before shifting toward exploitation.

The schedule balances the exploration-exploitation tradeoff critical to Q-learning convergence. Without sufficient exploration, the agent may never discover high-reward strategies; without eventual exploitation, it never capitalizes on learned knowledge.

• Initial ε: 1.0 (100% random)
• Final ε: 0.1 (10% random)
• Annealing period: 1 million frames
• Post-annealing: Constant 0.1 for continued mild exploration

The model-free reinforcement learning algorithm that forms the theoretical backbone of DQN. Q-Learning learns the value of taking a specific action in a given state by iteratively updating a Q-table using the Bellman equation. The core update rule is:

• Q(s,a) ← Q(s,a) + α[r + γ max Q(s',a') - Q(s,a)]
• Converges to the optimal action-value function Q* given sufficient exploration
• DQN extends this by replacing the tabular Q-table with a deep neural network function approximator

A critical DQN innovation that stores agent experiences as transition tuples (s, a, r, s') in a replay buffer and samples random mini-batches for training. This technique:

• Breaks temporal correlations between consecutive samples, satisfying i.i.d. assumptions of stochastic gradient descent
• Improves data efficiency by reusing each experience multiple times
• Smooths the learning distribution by averaging over many past behaviors
• Typical buffer sizes range from 100,000 to 1,000,000 transitions

A separate, periodically updated copy of the online Q-network used to compute stable TD targets. Without this mechanism, the Q-learning update becomes dangerously unstable because the same network is both predicting and generating the target value.

• Target network parameters θ⁻ are frozen for C steps
• Update rule: θ⁻ ← θ every C steps (e.g., every 10,000 iterations)
• Prevents harmful feedback loops where updates amplify oscillations
• This innovation was key to the 2015 DQN paper achieving superhuman Atari performance

The recursive mathematical decomposition that defines the optimal action-value function. It expresses the value of a state-action pair as the immediate reward plus the discounted value of the best possible next action:

• Optimality equation: Q*(s,a) = E[r + γ max_a' Q*(s',a') | s,a]
• γ (gamma) is the discount factor (typically 0.99), controlling the trade-off between immediate and future rewards
• DQNs minimize the Bellman error between predicted Q-values and Bellman targets
• Forms the theoretical foundation for all value-based RL methods

The learning paradigm that combines Monte Carlo methods (learning from actual experience) with dynamic programming (bootstrapping from current estimates). TD learning is the mechanism through which DQNs update their value estimates:

• TD(0) update: V(s) ← V(s) + α[r + γV(s') - V(s)]
• The term r + γV(s') - V(s) is the TD error, measuring the difference between predicted and actual returns
• Enables learning from incomplete episodes, unlike Monte Carlo methods
• Q-Learning is an off-policy TD control algorithm

The fundamental dilemma in sequential decision-making that DQNs must navigate. The agent must balance:

• Exploitation: Choosing the action with the highest current Q-value to maximize immediate reward
• Exploration: Trying suboptimal actions to discover potentially better strategies
• DQNs typically use an ε-greedy policy: with probability ε, take a random action; otherwise, act greedily
• ε is often annealed from 1.0 to 0.1 or 0.01 over training
• Poor exploration strategies lead to suboptimal convergence on local maxima