Trajectory Buffer

Unlike a standard experience replay buffer that stores individual transitions (state, action, reward, next state), a trajectory buffer stores entire episodes or trajectories. This is a contiguous sequence of tuples (s_t, a_t, r_t, s_{t+1}) from a start state to a terminal state. This structure is critical for algorithms that need to compute returns or analyze temporal dependencies across multiple steps.

• Example: Storing a full game of chess or a complete robotic manipulation task, rather than just individual moves.

A core function is to support on-policy reinforcement learning algorithms like PPO (Proximal Policy Optimization) and A2C (Advantage Actor-Critic). These algorithms require fresh data collected under the current policy for each update. The buffer temporarily holds these recent trajectories before they are used for a gradient step and then discarded or overwritten.

• Key Distinction: Contrast with off-policy buffers (e.g., in DQN) that store data from older policies for reuse over many updates.

By providing full sequences, the buffer enables accurate temporal credit assignment. Algorithms can compute Monte Carlo returns or n-step returns directly from the stored trajectory, which are essential for estimating the long-term value of actions.

• Mechanism: For a state s_t in a stored trajectory, the return G_t = r_t + γ*r_{t+1} + γ^2*r_{t+2} + ... can be calculated precisely, providing a low-bias learning signal.

Trajectory buffers are typically implemented as First-In-First-Out (FIFO) queues or circular buffers with a fixed capacity. When the buffer is full, the oldest complete trajectory is removed to make space for the newest one. This ensures the buffer contains data representative of the agent's most recent policy, which is vital for on-policy learning stability.

• Capacity Management: Size is measured in trajectories or total timesteps, balancing memory use with policy recency.

The sequential nature of trajectory data makes it the ideal training corpus for sequence modeling approaches to RL, such as the Decision Transformer. This architecture treats RL as a conditional sequence generation problem, where the model is trained on trajectories of (states, actions, returns) to predict future actions.

• Use Case: The buffer provides the structured (state, action, return-to-go) sequences required for autoregressive training of transformer-based policies.

It's crucial to distinguish a trajectory buffer from a standard transition-based replay buffer.

• Trajectory Buffer: Stores correlated sequences. Used for on-policy learning, policy gradient methods, and sequence modeling. Sampling is often done by selecting entire trajectories.
• Transition Buffer: Stores de-correlated individual experiences. Used for off-policy learning (e.g., DQN). Sampling is uniformly random or prioritized (PER) across all individual transitions to break temporal correlations.

Algorithms like REINFORCE, PPO (Proportionate Policy Optimization), and TRPO (Trust Region Policy Optimization) rely on trajectory buffers. They collect complete episodes of interaction—states, actions, rewards—and use them to compute policy gradient estimates. The buffer ensures the data used for an update is generated by the current policy, maintaining the on-policy assumption. Without storing full trajectories, these algorithms cannot accurately estimate the long-term return (cumulative reward) needed to assign credit to actions over time.

In agents like Dreamer and MuZero, a trajectory buffer stores sequences of real environment interactions. These sequences are used to train a world model—a neural network that predicts future states and rewards. The agent can then "imagine" or plan future trajectories entirely within this learned model. The buffer provides the necessary sequential, correlated data for the model to learn accurate temporal dynamics, which is impossible with uncorrelated, single-step transitions.

The Decision Transformer architecture treats reinforcement learning as a sequence modeling problem. It is trained on trajectories stored in a buffer, where each trajectory is a sequence of (state, action, return-to-go) tuples. The model learns to autoregressively generate actions conditioned on desired returns and past states. This paradigm shift from dynamic programming to pure supervised learning on trajectories relies entirely on a high-quality trajectory buffer for training data.

In continual learning, trajectory buffers manifest as episodic memory. Algorithms like Gradient Episodic Memory (GEM) store a subset of training examples (which can be sequential) from previous tasks. When learning a new task, the model computes gradients on the new data and the old data from the buffer. It then projects the new gradient to avoid increasing the loss on the buffered old examples, thereby mitigating catastrophic forgetting. The buffer preserves the temporal context of past tasks.

When learning from expert demonstrations (e.g., behavioral cloning), a demonstration buffer stores expert trajectories. The agent learns to map states to actions by supervised training on these sequences. Storing full trajectories is crucial for techniques like Dataset Aggregation (DAgger), where the agent interacts with the environment, queries an expert for corrective actions on visited states, and aggregates these new state-action pairs into the buffer to iteratively refine its policy.

While standard replay buffers can store single transitions, trajectory buffers are inherently structured for n-step learning. Algorithms use them to calculate n-step returns, which blend the accuracy of Monte Carlo returns with the lower variance of one-step Temporal Difference (TD) learning. By sampling a consecutive sequence of n steps from the buffer, the agent gets a more informed bootstrapping target, often leading to faster and more stable credit assignment compared to 1-step TD methods.