Deep Q-Network (DQN)

A DQN uses a deep neural network (typically a convolutional network) to approximate the optimal action-value function (Q-function). This network takes a high-dimensional state (e.g., raw pixels from a game screen) as input and outputs a vector of Q-values, one for each possible discrete action. This replaces the traditional Q-table, enabling generalization across a vast, continuous state space that would be infeasible to tabulate.

A replay buffer stores the agent's experiences as tuples (state, action, reward, next state, done). During training, mini-batches are sampled uniformly at random from this buffer.

• Breaks Temporal Correlations: Sequential experiences in RL are highly correlated; random sampling decorrelates them, leading to more stable gradient updates.
• Improves Data Efficiency: Each experience can be used for multiple weight updates.
• Smooths Training Distribution: Helps prevent the network from forgetting past experiences (catastrophic forgetting).

DQN employs a separate target network, which is a copy of the main Q-network. The target network's parameters are used to calculate the TD target (reward + γ * max_a Q_target(next_state, a)). These parameters are only updated periodically by copying the weights from the main network.

This innovation decouples the target from the parameters being optimized, addressing a core instability issue where the Q-value target shifts with every update, akin to "chasing a moving target." This leads to more consistent and convergent learning.

DQN is trained by minimizing the mean-squared error (MSE) between the current Q-value prediction and the computed TD target. The loss for a mini-batch is: L(θ) = E[(r + γ * max_a' Q_target(s', a'; θ⁻) - Q(s, a; θ))²]

Where:

• θ are the parameters of the online Q-network.
• θ⁻ are the parameters of the frozen target network.
• γ is the discount factor.

Standard backpropagation and gradient descent (e.g., RMSProp) are used to update θ. The use of a target network and experience replay makes this gradient-based optimization stable.

To handle partial observability in environments like Atari games, DQN stacks 4 consecutive grayscale frames (84x84 pixels) to form the state input. This provides a sense of motion and velocity.

Preprocessing steps are critical:

• Frame skipping: The agent selects an action every k frames (e.g., 4), repeating it in between. This speeds up simulation and reduces redundancy.
• Max-pooling over the last 2 frames: To handle flickering in some Atari games.
• These techniques convert the POMDP problem into a more Markovian state representation for the network.

DQN uses a simple ε-greedy policy to manage the exploration-exploitation tradeoff.

• With probability ε, the agent selects a random action (exploration).
• With probability 1-ε, the agent selects the action with the highest Q-value according to its current network (exploitation).

Typically, ε is annealed linearly from a high value (e.g., 1.0) to a low value (e.g., 0.1 or 0.01) over the course of training. This ensures heavy exploration early on, gradually shifting to greater reliance on the learned policy. More advanced variants replace this with noise-based exploration or intrinsic motivation.

Q-Learning is the foundational, tabular reinforcement learning algorithm upon which DQN is built. It learns an action-value function, Q(s,a), representing the expected cumulative reward for taking action a in state s and following the optimal policy thereafter.

• Core Mechanism: Iteratively updates Q-values using the Bellman equation: Q(s,a) ← Q(s,a) + α [r + γ max_a' Q(s',a') - Q(s,a)].
• Tabular Limitation: Requires discrete, enumerable state and action spaces, making it intractable for high-dimensional inputs like images from a robot's camera.
• Role for DQN: DQN's key innovation was using a deep neural network as a function approximator for this Q-table, enabling application to complex, continuous state spaces.

Experience Replay is a critical stabilization technique introduced with DQN. The agent stores its experiences (state, action, reward, next state, done) in a finite replay buffer and samples mini-batches from it randomly during training.

• Breaks Temporal Correlations: Sequential experiences in robotics are highly correlated (e.g., consecutive video frames). Random sampling decorrelates them, leading to more stable, i.i.d.-like training data.
• Improves Data Efficiency: Each experience can be used for multiple gradient updates, which is vital when robot interactions are slow or costly.
• Enables Off-Policy Learning: The buffer can contain experiences from older policies, allowing the algorithm to learn from past data, a hallmark of off-policy algorithms like DQN.

A Target Network is a second neural network, identical in architecture to the main Q-network, used to generate stable training targets. Its parameters are updated periodically by copying the main network's weights, rather than continuously.

• Mitigates Moving Target Problem: In standard Q-learning, the target r + γ max Q(s',a') depends on the same network being trained, causing destructive feedback loops and divergence. The target network provides a fixed, temporary target.
• Update Frequency: Originally updated every C steps (e.g., 10,000); modern variants like Double DQN and Soft Updates (polyak averaging) provide more sophisticated approaches.
• Impact on Convergence: This was, alongside experience replay, one of the two primary innovations that made stable deep RL possible.

The Bellman equation is the recursive optimality condition at the heart of most value-based RL algorithms, including DQN. It decomposes the value of a state (or state-action pair) into the immediate reward plus the discounted value of the successor state.

• Mathematical Foundation: For the optimal Q-function, it is expressed as: Q*(s,a) = E[ r + γ max_a' Q*(s', a') | s, a ].
• Bootstrapping: DQN uses this equation to create a temporal difference (TD) target. The network is trained to minimize the difference (TD error) between its prediction Q(s,a) and the target r + γ max Q_target(s',a').
• Planning vs. Learning: In dynamic programming, it's used for planning with a known model. In model-free RL like DQN, it's used for learning from sampled experience.

Double DQN is a seminal extension that addresses DQN's tendency to overestimate Q-values. It decouples the action selection from the action evaluation in the target calculation.

• The Overestimation Problem: The max operator in standard DQN max_a' Q_target(s', a') uses the same values to both select and evaluate an action, leading to an upward bias.
• Solution: Double DQN uses the online network to select the best action, and the target network to evaluate it: Target = r + γ Q_target(s', argmax_a' Q_online(s', a')).
• Impact: This simple modification reduces overoptimism, leads to more stable training, and typically improves final policy performance, making it a standard component in modern Q-learning variants.

Deep Deterministic Policy Gradient is a key actor-critic algorithm that extends the DQN philosophy to continuous action spaces, which are ubiquitous in robotics (e.g., joint torques, wheel velocities).

• Actor-Critic Architecture: Employs two networks: an Actor (policy) that maps states to precise actions, and a Critic (Q-function) that evaluates those actions, similar to DQN.
• DQN Heritage: It incorporates experience replay and target networks for both actor and critic networks, directly inheriting DQN's stabilization techniques.
• Deterministic Policy: Outputs a specific action rather than a distribution, requiring added noise (e.g., Ornstein-Uhlenbeck) to the actor's output during training for exploration. It is a foundational algorithm for robotic arm manipulation and locomotion.