Decision Transformer

The model is conditioned on a Return-to-Go (RTG) token at each timestep, which represents the cumulative reward desired from that point forward. This transforms goal achievement into a conditional generation task. The model learns to output actions that fulfill the specified RTG trajectory.

• Training: The RTG for a timestep is calculated as the sum of future rewards in the trajectory.
• Inference: A user specifies an initial high RTG (the desired total reward), and the model autoregressively generates actions to achieve it.

It leverages a causal Transformer decoder architecture (like GPT) to model the joint distribution of a trajectory sequence. The model processes a concatenated sequence of states, actions, and returns.

• Input Format: The sequence is ordered as: RTG₁, state₁, action₁, RTG₂, state₂, action₂, ...
• Causal Attention: Ensures predictions for actionₜ depend only on past and present states/RTGs, not future ones, preserving the autoregressive property.
• Benefits: Inherits the Transformer's ability to capture long-range dependencies in trajectories, which is crucial for credit assignment in long-horizon tasks.

The Decision Transformer is an offline RL algorithm. It is trained on a fixed dataset of trajectories collected by some (potentially sub-optimal) behavioral policy, stored in a trajectory buffer. It does not interact with the environment during training.

• Objective: Maximize the log-likelihood of the actions in the dataset, conditioned on the states and RTGs.
• Advantage: Avoids the instability and exploration challenges of online RL. It can extract high-performance policies from sub-optimal or heterogeneous data.
• Limitation: Performance is inherently bounded by the quality and coverage of the offline dataset.

Actions are generated autoregressively, one timestep at a time, similar to text generation in language models. The model predicts the next action token based on the history of RTGs, states, and previously generated actions.

• Process: Given an initial state and target RTG, the model predicts action₁. This action is appended to the sequence, the environment returns the next state, the RTG is decremented by the received reward, and the process repeats.
• Implication: The model implicitly learns a goal-conditioned policy π(action | state, desired return).

Unlike value-based methods (e.g., DQN) or actor-critic methods, the Decision Transformer does not explicitly perform dynamic programming or Bellman backups. It avoids learning a Q-function or value function.

• Mechanism: Learning is driven purely by supervised sequence prediction on the offline dataset.
• Benefit: Eliminates the challenges of temporal difference (TD) error propagation, bootstrapping, and moving target networks, which are common sources of instability in deep RL.
• Trade-off: While more stable, it may not perform as well as model-free RL in online fine-tuning settings where efficient credit assignment via TD learning is critical.

The model's attention mechanism over the input sequence provides a flexible mechanism for credit assignment. It can learn to associate past states and actions with changes in the RTG signal.

• Function: The model can attend to key decision points earlier in the trajectory to understand their long-term consequences on the return.
• Analogy: Similar to a language model using context to determine the next word, the Decision Transformer uses trajectory context to determine the next optimal action.
• Limitation: Effective credit assignment is constrained by the fixed context length of the Transformer, which may not span extremely long horizons.

A fixed-size data structure, typically a circular buffer, that stores past experiences (state, action, reward, next state) for off-policy reinforcement learning. By randomly sampling mini-batches from this buffer, algorithms break harmful temporal correlations in sequential data, stabilize training, and dramatically improve sample efficiency. It is the foundational component used by DQN and most modern RL agents.

A specialized replay buffer that stores and samples complete sequences (trajectories) of states, actions, and returns, rather than individual transitions. This is essential for training sequence models like the Decision Transformer, which requires the full temporal context of a past episode to model returns-to-go and generate future actions. It contrasts with standard buffers built for value-based methods like DQN.

The primary learning signal in value-based reinforcement learning, calculated as the difference between the current value estimate and a bootstrapped target. It drives credit assignment, indicating how 'surprising' a transition was.

• Key Role in PER: In Prioritized Experience Replay (PER), the absolute TD error is used to prioritize sampling, focusing learning on more informative experiences.
• Contrast with DT: The Decision Transformer sidesteps explicit TD error by framing RL as a conditional sequence modeling problem, predicting actions given a desired return.

A family of algorithmic techniques that correct for the discrepancy between the behavior policy (which collected the data) and the target policy (being learned). Methods like V-trace and Retrace adjust update targets to guarantee convergence in off-policy learning. The Decision Transformer is inherently off-policy—it learns from trajectories in its replay buffer generated by any policy—but it avoids explicit correction by using a supervised sequence loss on the stored data.

A continual learning technique where a generative model (e.g., a GAN or VAE) is trained to produce synthetic data samples from previous tasks. These generated samples are interleaved with new task data during training to mitigate catastrophic forgetting. This shares a conceptual link with model-based experience replay, where a learned world model generates synthetic experiences for reinforcement learning.

An advanced replay paradigm where a learned world model (a neural network that predicts state transitions and rewards) is used to generate synthetic experiences. Algorithms like Dreamer and MuZero train their policies largely on these imagined trajectories, improving sample efficiency and enabling planning. This contrasts with the Decision Transformer, which is a model-free method that replays real stored trajectories without learning an explicit dynamics model.