Batch Sampling

Batch sampling provides a stochastic approximation of the true gradient over the entire dataset. By randomly selecting a small subset of experiences (e.g., 32-512 transitions), it:

• Reduces computational cost per update compared to full-batch gradient descent.
• Introduces noise that can help escape shallow local minima.
• Enables online learning by decoupling the data generation process (interacting with the environment) from the optimization process.

Sequential experiences in reinforcement learning are highly correlated. Batch sampling from a large, shuffled replay buffer is critical to:

• I.I.D. Assumption: Provide the independent and identically distributed data samples assumed by most stochastic optimization algorithms.
• Stabilize Training: Prevent the feedback loops and destructive policy oscillations that occur when training on highly correlated sequential data.
• This decorrelation is a primary reason experience replay was essential for stabilizing Deep Q-Networks (DQN).

The sampling strategy is dictated by the underlying buffer data structure:

• Uniform Sampling (Circular Buffer): The simplest method. Experiences are stored in a fixed-size FIFO queue; sampling is uniform random. Oldest data is overwritten when full.
• Prioritized Sampling: Experiences are sampled with probability proportional to their Temporal-Difference (TD) error, focusing learning on surprising or informative transitions. Requires a priority queue (e.g., a SumTree).
• Reservoir Sampling: Used for streams of unknown length, maintaining a statistically uniform random sample without knowing the total data size in advance.

Batch sampling behavior is controlled by critical hyperparameters:

• Batch Size: The number of transitions sampled per update. A larger batch gives a lower-variance gradient estimate but increases memory and compute. Typical range: 32 to 1024.
• Buffer Capacity: The maximum number of experiences stored. A larger capacity increases diversity but may retain outdated data from earlier, poorer policies.
• Replay Ratio: The number of gradient updates performed per new environment step. A ratio > 1 means the agent "replays" past data more than it collects new data, improving sample efficiency.

Batch sampling is the engine of off-policy reinforcement learning algorithms like DQN, DDPG, and SAC. It allows:

• Policy Decoupling: The behavior policy that collects data can differ from the target policy being optimized. Old experiences from any policy remain valid for learning.
• Data Reuse: High-value experiences can be used for multiple gradient updates, dramatically improving sample efficiency.
• Importance Sampling: When the behavior and target policies differ significantly, importance sampling ratios can be applied to the sampled batch to correct for distributional bias.

Beyond uniform random sampling, advanced strategies tailor the learning process:

• Hindsight Experience Replay (HER): For goal-conditioned tasks, re-samples failed episodes by relabeling them with achieved goals, learning from failure.
• Trajectory Sampling: Samples entire sequences (trajectories) instead of individual transitions, crucial for training recurrent policies or algorithms like Decision Transformers.
• Model-Based Augmentation: Uses a learned world model (e.g., in Dreamer) to generate synthetic experiences, which are added to the buffer to improve planning and exploration.

The Experience Replay Buffer is the foundational data structure that stores past interactions (state, action, reward, next state) for later sampling. It serves two primary purposes:

• Breaks temporal correlations by randomizing the order of training data.
• Improves sample efficiency by allowing the reuse of experiences. Common implementations include the Circular Buffer (FIFO) and strategies like Reservoir Sampling for streaming data. Its Buffer Capacity is a critical hyperparameter balancing memory and data diversity.

Prioritized Experience Replay is an advanced sampling strategy that selects transitions with a probability proportional to their Temporal Difference (TD) Error. This focuses learning on experiences the agent found most surprising or informative.

• Uses a stochastic prioritization to ensure all experiences are eventually visited.
• Requires Importance Sampling weights to correct the bias introduced by non-uniform sampling, ensuring convergence. It is a key component in algorithms like Rainbow DQN.

Off-Policy Learning is the paradigm where an agent learns a target policy from data generated by a different behavior policy. Experience replay is inherently off-policy.

• Importance Sampling is the core statistical technique for correcting the distribution mismatch between policies.
• Off-Policy Correction algorithms like V-trace and Retrace provide more stable, lower-variance targets for value function updates when using older data. This enables learning from exploratory data or historical datasets.

While batch sampling often uses single transitions, some algorithms require sequences.

• A Trajectory Buffer stores and samples complete episodes, crucial for on-policy algorithms or models like the Decision Transformer.
• N-Step Returns unify Monte Carlo and TD methods by bootstrapping after N steps. Batch sampling from a replay buffer must handle these multi-step transitions, which provide a better bias-variance tradeoff for learning value functions.

In continual learning, replay buffers are used to mitigate Catastrophic Interference.

• Gradient Episodic Memory (GEM) and its efficient variant A-GEM store past examples and constrain new gradients to not increase loss on them.
• Generative Replay uses a generative model to produce synthetic past data for interleaving with new tasks. These methods use batch sampling from a memory of past experiences to preserve knowledge, directly linking experience replay to continual learning systems.

Here, replay buffers train a World Model—a neural network that predicts future states and rewards.

• Dreamer agents learn a policy entirely within the latent imagination of their world model, with experiences replayed from the model.
• MuZero learns a model implicitly for planning and uses a replay buffer of real trajectories to train its networks. Batch sampling provides the diverse data needed to learn accurate dynamics models, enabling more data-efficient planning.