Deep Q-Network (DQN)

Replaces the traditional Q-table with a deep convolutional or feedforward network. Instead of storing a value for every state-action pair—impossible in continuous or high-dimensional state spaces like raw pixel data or sensor telemetry—the network generalizes across similar states. The input is the environment's state representation (e.g., a warehouse occupancy grid), and the output layer has one neuron per possible action, each predicting the expected cumulative reward. This allows the agent to handle the curse of dimensionality inherent in logistics state spaces.

A memory bank storing the agent's transitions as tuples: (state, action, reward, next_state, done). During training, mini-batches are sampled uniformly at random from this buffer rather than using consecutive experiences. This breaks the strong temporal correlations in sequential data, satisfying the i.i.d. assumption of stochastic gradient descent. It also allows rare but critical experiences—like a sudden stockout or route disruption—to be replayed multiple times, dramatically improving sample efficiency.

DQN uses two networks with identical architecture: a policy network (updated every step) and a target network (updated periodically). The target network generates the TD target values used in the loss calculation. By keeping these targets fixed for several thousand steps, the algorithm prevents a destructive feedback loop where the policy chases a constantly shifting target. This stabilization is critical for convergence in non-stationary logistics environments like dynamic fleet routing.

The network is trained to minimize the mean squared error between its predicted Q-value and the Temporal Difference target: R + γ * max_a' Q_target(s', a'). The TD target bootstraps the current estimate from the next state's value, enabling learning from incomplete episodes. The loss gradient is backpropagated only through the policy network. This mechanism allows the agent to learn optimal inventory replenishment policies without waiting for end-of-episode outcomes.

Balances the exploration-exploitation trade-off by selecting a random action with probability ε, and the greedy action (argmax Q-value) otherwise. Epsilon is typically annealed from 1.0 to a small final value like 0.01 over millions of frames. This ensures the agent thoroughly explores the state space early in training—critical for discovering novel logistics strategies—before committing to the learned policy. Advanced variants replace this with Boltzmann exploration or parameter-space noise for more structured exploration.

An architectural refinement that addresses the overestimation bias of standard DQN. The policy network selects the best action for the next state, but the target network evaluates its value. This decouples action selection from evaluation: R + γ * Q_target(s', argmax_a' Q_policy(s', a')). In supply chain contexts with noisy rewards—like stochastic demand signals—this prevents the agent from developing an overly optimistic view of risky inventory positions.

The foundational model-free, off-policy algorithm that DQN extends. It learns an action-value function Q(s,a) that estimates the total reward an agent can expect after taking action a in state s. The core update rule uses the Bellman equation to bootstrap from the maximum Q-value of the next state. DQN replaces the tabular Q-table with a deep neural network to handle high-dimensional state spaces like warehouse sensor data.

A critical DQN innovation that stores agent transitions (state, action, reward, next state) in a replay buffer. During training, random mini-batches are sampled from this buffer rather than using consecutive experiences. This breaks the harmful temporal correlations between sequential samples and allows rare, high-value experiences to be reused multiple times, dramatically improving data efficiency and training stability in logistics simulations.

DQN uses two neural networks: a policy network that is updated every step and a target network that is a frozen copy updated periodically. The target network generates the Q-value targets for the Bellman update. This separation prevents a destructive feedback loop where the network chases a moving target, a phenomenon known as non-stationarity. The target network update frequency is a key hyperparameter for convergence.

The mathematical framework that formalizes the environment DQN interacts with. An MDP is defined by a tuple (S, A, P, R, γ)—states, actions, transition probabilities, rewards, and a discount factor. The Markov property assumes the future depends only on the current state, not the history. Logistics problems like inventory routing are modeled as MDPs where DQN learns optimal sequential decisions.

DQN agents must balance exploiting known high-value actions with exploring unknown actions that might yield even higher returns. The standard approach is an ε-greedy policy: with probability ε, a random action is chosen; otherwise, the action with the highest Q-value is selected. Epsilon is typically annealed from 1.0 to a small value like 0.01 over training, ensuring broad exploration early and precise exploitation later.

A more realistic extension of the MDP where the agent cannot directly observe the full state. Instead, it receives observations that provide partial, noisy information. To handle this, DQNs are often augmented with recurrent layers (creating a DRQN) that maintain an internal hidden state over time, effectively integrating past observations to infer the true underlying state. This is critical when sensor data is incomplete.