Experience Replay

A replay buffer (or replay memory) is a fixed-size data structure that stores agent experiences as tuples: (state, action, reward, next_state, done). The buffer operates as a circular queue—when full, the oldest transitions are overwritten. This finite capacity ensures the agent learns from recent, relevant experiences while discarding obsolete data. The buffer size is a critical hyperparameter; too small and the agent overfits to recent trajectories, too large and training becomes computationally expensive with stale policies.

The core innovation of experience replay is uniform random sampling from the buffer. By drawing mini-batches randomly rather than sequentially, the technique breaks the strong temporal correlations present in consecutive experiences. This satisfies the i.i.d. (independent and identically distributed) assumption that stochastic gradient descent requires for stable convergence. Without this decorrelation, deep neural networks used as function approximators in algorithms like Deep Q-Networks (DQN) would diverge or oscillate wildly during training.

An enhancement over uniform sampling, Prioritized Experience Replay (PER) assigns a priority score to each transition based on its Temporal Difference (TD) error—the difference between predicted and target Q-values. Transitions with higher TD errors are sampled more frequently because they represent experiences where the agent's predictions were most incorrect. This prioritization accelerates learning by focusing computation on surprising or high-information events. To correct the introduced sampling bias, PER uses importance-sampling weights during the gradient update.

In online reinforcement learning, each experience is used once for a gradient update and then discarded. Experience replay enables multiple gradient updates per collected transition, dramatically improving sample efficiency. This is critical in real-world logistics applications—such as warehouse robot navigation or fleet routing—where generating experiences is expensive, time-consuming, or involves physical risk. An agent can replay a single successful docking maneuver hundreds of times, extracting maximum learning value from scarce real-world interactions.

Experience replay is the key enabler of off-policy learning, where an agent learns from data generated by a different (often older) policy. The buffer stores transitions from many previous iterations of the agent's behavior policy, allowing the current target policy to learn from a diverse distribution of experiences. This is essential for algorithms like DQN, which maintain a separate target network updated less frequently. Off-policy learning with replay buffers also permits learning from demonstrations or historical logs, not just live agent interaction.

Neural networks are prone to catastrophic forgetting—the tendency to overwrite previously learned knowledge when trained on new, correlated data. By maintaining a diverse buffer of past experiences and interleaving them with new transitions during training, experience replay acts as a form of continual learning. The agent rehearses rare but important edge cases (e.g., a supply chain disruption event) alongside common scenarios, preventing the policy from collapsing to a narrow set of behaviors that only work in the most recent environment conditions.

The seminal architecture that popularized experience replay for reinforcement learning. A DQN uses a deep neural network to approximate the Q-value function and stores transitions in a replay buffer. By sampling random mini-batches from this buffer, the algorithm breaks the temporal correlations between consecutive samples, satisfying the i.i.d. assumption of stochastic gradient descent. This prevents the network from overfitting to recent experiences and catastrophic forgetting.

An enhancement to uniform sampling that assigns higher probability to transitions with high temporal-difference (TD) error. The intuition is that experiences where the agent's prediction was significantly wrong contain more learning potential. To avoid bias, importance-sampling weights correct the introduced distribution shift. This dramatically accelerates learning in environments with sparse rewards by replaying surprising or high-value transitions more frequently.

A technique designed for sparse reward environments where an agent rarely achieves the goal. HER stores not only the actual trajectory but also replays it with a virtual goal—the state the agent actually reached. This reframes failure as success for a different objective, allowing the agent to learn from every episode regardless of the outcome. It is critical for robotics and logistics tasks where successful completions are initially rare.

The phenomenon where a neural network abruptly overwrites previously learned knowledge when trained on new data. In online RL, sequential training on correlated experiences causes this degradation. Experience replay mitigates this by interleaving old and new experiences during training. The buffer acts as a long-term memory store, ensuring the agent rehearses past strategies and maintains a stable value function across the entire state distribution.

A learning paradigm where the behavior policy (used to collect data) differs from the target policy (being optimized). Experience replay is a natural fit for off-policy algorithms like Q-Learning because the buffer can store trajectories generated by older policies. The agent learns the optimal policy by reusing data collected from exploratory or suboptimal behaviors, maximizing data efficiency in logistics simulations where data generation is expensive.

The data structure implementing the sliding window of agent experiences. Key design choices include:

• Capacity: Limits memory usage by evicting oldest transitions (FIFO)
• Sampling Strategy: Uniform, prioritized, or stratified
• Storage Format: Typically stores tuples of (state, action, reward, next_state, done) In distributed logistics systems, this buffer may be sharded across multiple workers to handle high-throughput agent interactions.