Buffer Capacity

The primary trade-off in setting buffer capacity is between memory constraints and the diversity of experiences. A larger buffer stores a more representative sample of the state-action space, which decorrelates updates and improves learning stability. However, it increases memory overhead and can slow down sampling. A small buffer is memory-efficient but may lack sufficient diversity, causing the agent to overfit to recent, potentially correlated transitions.

• Large Capacity (>1M transitions): Used in complex environments (e.g., Atari games) to ensure sufficient coverage.
• Small Capacity (10k-100k): Used in simpler tasks or when memory is a bottleneck, such as on edge devices.

In continual learning and non-stationary environments, buffer capacity directly influences the rate of catastrophic forgetting. A buffer that is too small will rapidly overwrite old experiences, causing the agent to lose knowledge of infrequent but important states. A sufficiently large buffer acts as an episodic memory, preserving a core set of past experiences that can be replayed to consolidate old knowledge while learning new tasks.

• Fixed-Capacity Circular Buffer: The standard FIFO approach; old data is permanently lost once overwritten.
• Reservoir Sampling: An alternative for streams, maintaining a statistically uniform sample of all past data, but is less common in deep RL due to implementation complexity.

Buffer capacity's effectiveness is modulated by the sampling strategy. With uniform random sampling, a large buffer ensures a diverse mini-batch. With Prioritized Experience Replay (PER), capacity influences the distribution of Temporal-Difference (TD) errors stored. A small buffer may not retain low-priority (low-error) experiences needed for a balanced learning signal.

• PER with Large Buffer: Maintains a long-tail distribution of TD errors, enabling focus on both recent surprises and old, stable knowledge.
• Replay Ratio: This hyperparameter, defining updates per new experience, must be tuned relative to capacity to avoid overfitting to a stale buffer.

Buffer capacity has direct computational consequences. Insertion and sampling operations must scale efficiently. A common implementation is a circular buffer using contiguous memory (e.g., a NumPy array).

• Sampling Complexity: O(1) for random index access, but batch sampling from a very large buffer can cause memory bandwidth issues.
• Hardware Limits: On GPUs, experiences are often stored in host (CPU) RAM. Transferring large mini-batches from a massive buffer can become a training bottleneck.
• Rule of Thumb: Capacity is often set to an order of magnitude larger than the mini-batch size (e.g., 10x to 1000x) to ensure sufficient randomness.

Optimal capacity is highly environment-dependent. Key factors include:

• Episode Length: For long episodes (e.g., robotics), capacity must hold multiple complete trajectories to learn temporal credit assignment.
• State Dimensionality: High-dimensional states (e.g., images) consume more memory per transition, often forcing a practical limit on total capacity.
• Non-Stationarity: In environments where the reward function or dynamics change (concept drift), a larger buffer helps the agent detect drift by comparing recent and old experiences.

Empirical Starting Point: Many seminal papers (e.g., DQN on Atari) use a capacity of 1,000,000 transitions as a robust default for pixel-based tasks.

Advanced implementations move beyond fixed capacity. Dynamic buffers can grow, shrink, or prioritize retention based on learned metrics.

• Quality-Aware Retention: Integrates with PER to not just sample by priority, but to decide which experiences to keep based on estimated utility, potentially overwriting low-value data even if the buffer isn't full.
• Task-Conditional Buffers: In continual learning, separate buffers or buffer partitions per task can be maintained, with capacity allocated based on task complexity or importance.
• Theoretical Limit: In the infinite-capacity limit with uniform sampling, the buffer approximates the entire stationary data distribution, turning online RL into a form of offline RL.

The most common pattern, implemented as a circular buffer or ring buffer. When the buffer reaches its defined capacity, the oldest experience tuple is overwritten by the newest one. This ensures a bounded memory footprint and naturally prioritizes recent data, which is critical in non-stationary environments.

• Advantage: Simple, memory-efficient, and provides implicit handling of distribution shift.
• Trade-off: Can lead to catastrophic forgetting if important, rare experiences are prematurely ejected.
• Example: The original Deep Q-Network (DQN) used a fixed-size replay buffer of 1 million transitions.

Used when learning from an infinite or very long data stream of unknown length. This algorithm maintains a buffer of fixed capacity N and ensures that every item seen in the stream has an equal probability (N / stream_length) of being in the final buffer. It's statistically optimal for maintaining a uniform random sample.

• Use Case: Ideal for online learning architectures and continual learning where data arrives sequentially and the underlying distribution may change.
• Implementation: For each new experience i, it is added to the buffer with probability N/i. If added, a random existing experience is evicted.

Integrates buffer capacity with Prioritized Experience Replay (PER). While the total capacity is fixed, the eviction policy is not simply FIFO. Experiences are stored with a priority (e.g., based on Temporal-Difference (TD) error). Sampling is biased toward high-priority experiences, and eviction may also target low-priority items.

• Challenge: Requires maintaining a priority heap or sum-tree, adding computational overhead.
• Benefit: Maximizes the informational value per unit of buffer capacity, accelerating learning.

The buffer's capacity is not fixed but can grow, shrink, or be partitioned based on learning signals. This is a more advanced pattern used in complex continual learning systems.

• Partitioned Buffers: Allocate separate segments for different tasks, modes, or data distributions to combat interference.
• Growth Strategies: Start with a small buffer and expand it until a memory budget is reached, allowing early learning to be more sample-efficient.
• Eviction by Utility: Uses a heuristic (e.g., age, priority, density) to decide which experience to remove, potentially preserving a more diverse set.

Augments a finite experience replay buffer with a generative model. The core buffer holds a curated set of real experiences up to its capacity. A separately trained generative model (e.g., a GAN or VAE) can then produce synthetic replays of past data on demand.

• Purpose: Effectively provides "infinite" capacity for past data distributions, directly addressing catastrophic forgetting.
• Relation: This is the core mechanism behind Generative Replay in continual learning.
• Consideration: Adds the complexity and training cost of maintaining a high-fidelity generative model.

Buffer capacity is not set in isolation; it is tuned in conjunction with the replay ratio. This hyperparameter defines how many times, on average, an experience is reused before being evicted.

• Formula: Average Reuses = (Buffer Capacity * Replay Ratio) / Batch Size.
• Small Buffer, High Ratio: Experiences are reused heavily, leading to overfitting and potential bias.
• Large Buffer, Low Ratio: Provides high diversity but may include stale, irrelevant data, slowing learning.
• Optimal Point: Found empirically, balancing sample efficiency with training stability and adaptation speed.

The foundational data structure that stores past experience tuples (state, action, reward, next state). Its primary functions are:

• Decorrelation: Breaking temporal correlations by random sampling.
• Data Reuse: Increasing sample efficiency by learning from transitions multiple times.
• Stabilization: Providing a more stationary distribution for training than the online stream. The buffer's capacity directly determines the size and diversity of this stored dataset.

An advanced sampling strategy that selects experiences based on their estimated learning potential, rather than uniformly. It uses the Temporal-Difference (TD) error as a priority signal.

• High TD Error: Indicates a 'surprising' transition where the agent's prediction was wrong; prioritized for faster learning.
• Stochastic Prioritization: Introduces randomness to ensure all experiences are eventually sampled.
• Importance Sampling: Corrects bias introduced by the non-uniform sampling distribution. Buffer capacity influences the distribution of priorities and the effectiveness of importance sampling corrections.

The tendency of a neural network to abruptly lose previously learned knowledge when trained on new data. Experience replay is a primary defense mechanism.

• Buffer as Episodic Memory: Stores exemplars from past tasks/distributions.
• Interleaved Training: By mixing old (replayed) and new data in mini-batches, the model is forced to retain performance on both.
• Capacity Trade-off: A larger buffer can hold more diverse past experiences, providing stronger protection against forgetting, but at increased memory cost.

A critical hyperparameter defined as the average number of gradient updates performed using a replayed experience before it is discarded. It governs the relationship between new and old data.

• Formula: (Updates per step) ≈ (Batch Size) * (Replay Ratio) / (Batch Size).
• High Ratio: The agent learns more from each stored transition, favoring data efficiency and stability.
• Low Ratio: The agent adapts more quickly to new data, favoring plasticity. Buffer capacity and replay ratio are co-designed: a small buffer requires a high ratio to reuse data sufficiently, while a large buffer can operate with a lower ratio.

A learning paradigm where the policy being improved (the target policy) differs from the policy used to generate behavior (the behavior policy). The replay buffer is essential for this.

• Data Decoupling: Stores experiences from old behavior policies.
• Policy Evaluation: Allows learning the value of a new policy using data generated by a different policy.
• Capacity Impact: A larger buffer capacity increases the diversity of behavior policies represented, improving the robustness of off-policy evaluation and control for the current target policy.

The most common low-level implementation for a fixed-capacity experience replay buffer. It is a First-In-First-Out (FIFO) queue with constant-time insertion and sampling.

• Fixed Array: Pre-allocates memory for the maximum capacity.
• Head/Tail Pointers: Tracks the oldest and newest entries.
• Automatic Overwrite: When full, adding a new experience overwrites the oldest one. This implementation directly enforces the buffer capacity constraint and is computationally efficient, making it the standard for large-scale deep reinforcement learning.