Deep Q-Network (DQN)

A replay buffer stores the agent's experiences (state, action, reward, next state, done) as they interact with the environment. During training, mini-batches are sampled randomly from this buffer. This technique:

• Breaks temporal correlations between sequential experiences, which can lead to unstable feedback loops.
• Increases data efficiency by allowing each experience to be used in multiple weight updates.
• Smooths training by averaging over many past states, reducing variance in the learning updates.

DQN uses a separate, periodically updated network to calculate the Q-targets in the Bellman equation. This target Q-network is a copy of the main online network. Its parameters are frozen and only updated every C steps by copying the online network's weights. This decouples the target from the parameters being optimized, addressing a moving target problem. Without it, the Q-values the network is trying to converge toward would shift with every update, causing severe instability and divergence.

To handle the partial observability of raw pixel inputs (e.g., a single Atari frame doesn't show velocity), DQN stacks the last 4 grayscale, downsampled frames to form the state input. This provides a sense of motion and object dynamics. A standardized preprocessing pipeline (cropping, grayscaling, resizing to 84x84) is applied to all frames, reducing input dimensionality and computational cost while maintaining essential visual information for the convolutional neural network.

DQN minimizes the mean squared error (MSE) between the current Q-value prediction and the temporal difference (TD) target. The loss for a transition is: L(θ) = (r + γ * max_a' Q_target(s', a'; θ-) - Q(s, a; θ))² where θ are the online network parameters and θ- are the target network parameters. To further stabilize training, gradients are often clipped to a maximum norm (e.g., 10) before the update step. This prevents exploding gradients that can occur in deep networks with recursive Bellman updates.

DQN employs a simple but effective epsilon-greedy strategy to manage the exploration-exploitation tradeoff. During training:

• With probability ε, the agent selects a random action (exploration).
• With probability 1-ε, the agent selects the action with the highest Q-value according to the online network (exploitation). The value of ε is typically annealed linearly from a high value (e.g., 1.0) to a low value (e.g., 0.01 or 0.1) over the course of training. This ensures extensive early exploration of the state-action space, gradually shifting toward exploitation of the learned policy.

The original DQN used a convolutional neural network (CNN) to process raw pixel inputs. The architecture consisted of:

• Three convolutional layers with ReLU activations to extract spatial features.
• Two fully connected layers culminating in a linear output layer with one node per possible action. This design allows the network to learn hierarchical visual representations directly from pixels, translating them into state-action value estimates. The network outputs the predicted Q-value for all actions in a given state in a single forward pass, enabling efficient action selection via a simple argmax operation.

Q-Learning is the foundational, model-free, off-policy algorithm upon which DQN is built. It learns the optimal action-value function (Q-function) by iteratively applying the Bellman equation. The core update rule is: Q(s,a) = Q(s,a) + α * [r + γ * max_a' Q(s',a') - Q(s,a)]. DQN's primary innovation is using a deep neural network to approximate this Q-function for high-dimensional state spaces where a tabular representation is impossible.

Experience Replay is a critical stabilization technique introduced with DQN. The agent stores its experiences (state, action, reward, next state, done) in a finite replay buffer. During training, it samples mini-batches of experiences randomly from this buffer.

Key benefits include:

• Breaks temporal correlations between sequential samples, improving learning stability.
• Increases data efficiency by reusing past experiences multiple times.
• Enables off-policy learning, allowing the use of experiences collected under older policies.

Temporal Difference (TD) Learning is the family of algorithms that combine ideas from Monte Carlo methods and dynamic programming. DQN is a TD learning algorithm. It learns by updating its value estimate based on the difference (the TD error) between its current prediction and a more informed estimate—the TD target (r + γ * max Q(s',a')). This bootstrapping approach allows for online learning without waiting for an episode to finish, making it efficient for continuous tasks.

The Bellman equation provides the recursive, mathematical foundation for value-based RL algorithms like DQN. For the optimal action-value function Q*, it is expressed as: Q*(s,a) = E[ r + γ * max_a' Q*(s', a') ]. DQN's training objective is to minimize the difference between its network's output and the target derived from this equation (the TD target). This minimization is performed via gradient descent on the Mean Squared Bellman Error (MSBE).

The Target Network is a second neural network, identical in architecture to the main Q-network, used to calculate stable TD targets. Its parameters are periodically copied from the main network (e.g., every C steps) or softly updated via polyak averaging. This prevents a moving target problem, where the Q-values the network is trying to converge to shift with every gradient step, which can lead to catastrophic divergence. It is a cornerstone of stable deep RL.

The exploration-exploitation tradeoff is the fundamental dilemma in RL. An agent must exploit known rewarding actions while also exploring unknown actions to potentially discover better strategies. DQN typically uses an ε-greedy policy for action selection, where the agent selects a random action with probability ε (exploration) and the action with the highest Q-value with probability 1-ε (exploitation). ε is often annealed from a high value (e.g., 1.0) to a low value (e.g., 0.01 or 0.1) over time.