Experience Replay

In sequential decision-making, consecutive transitions are highly correlated, which violates the independent and identically distributed (i.i.d.) assumption of stochastic gradient descent. By storing transitions in a replay buffer and uniformly sampling random mini-batches, experience replay shatters these harmful temporal correlations, leading to more stable and convergent neural network training.

Without a replay buffer, each experience is used once for a single gradient update and then discarded—an extremely inefficient use of data in sparse reward environments. Experience replay allows each transition tuple (state, action, reward, next state) to be sampled and learned from multiple times, dramatically improving sample efficiency and reducing the total number of interactions required with the environment.

Standard uniform sampling treats all transitions equally, but some experiences are more surprising or instructive than others. Prioritized Experience Replay assigns a sampling probability proportional to the temporal-difference (TD) error magnitude. Transitions with larger prediction errors are replayed more frequently, accelerating learning by focusing the agent's attention on the most salient corrective feedback.

The replay buffer is typically implemented as a circular buffer with a fixed maximum capacity. When full, the oldest transitions are overwritten, ensuring the agent gradually forgets outdated experiences from an obsolete policy. Key design parameters include:

• Buffer capacity: Balances memory constraints against the diversity of stored experiences
• Sampling strategy: Uniform, prioritized, or hindsight-based
• Storage format: Raw observations or compressed latent features

Experience replay is the critical enabler of off-policy learning, where the policy being updated differs from the policy that generated the data. By learning from a mixture of transitions produced by older versions of the policy, the agent can extract generalizable knowledge without requiring fresh on-policy samples. This decoupling is essential for algorithms like Deep Q-Networks (DQN) and Soft Actor-Critic (SAC).

Experience replay works in tandem with a target network—a frozen copy of the main Q-network updated periodically. Without this pairing, the moving target problem causes destructive feedback loops during bootstrapping. The replay buffer provides a stable, decorrelated dataset, while the target network provides consistent TD targets, together preventing the policy from chasing its own rapidly shifting predictions.

A failure mode in neural networks where learning new information abruptly overwrites previously acquired knowledge. Experience replay mitigates this by interleaving old and new experiences during training. By constantly revisiting past transitions, the network maintains performance across the entire distribution of states rather than overfitting to the most recent trajectory. This is critical in non-stationary environments like wireless networks where traffic patterns shift.

A reinforcement learning paradigm where the behavior policy (used to collect data) differs from the target policy (being optimized). Experience replay is inherently off-policy because the buffer contains transitions generated by older versions of the policy. Algorithms like Q-learning and SAC exploit this property to reuse data many times, dramatically improving sample efficiency compared to on-policy methods like PPO that require fresh trajectories.