Autoregressive Decoding

The core mechanism where each predicted token is conditioned on all previously generated tokens. This creates a strict temporal dependency, making the generation of action t_n a function of the sequence [t_1, t_2, ..., t_{n-1}]. This is enforced by a causal mask in the transformer's attention mechanism, which prevents the model from 'seeing' future tokens during training and inference.

• Implication for Robotics: Ensures that a robot's current movement logically follows from its prior actions, maintaining temporal coherence in tasks like 'pick, then place'.

A specific attention mask applied in transformer decoder layers to enforce the autoregressive property. It allows a token to attend only to itself and preceding tokens in the sequence, creating a triangular mask of ones and negative infinities.

• Technical Role: Enables parallel training (teacher forcing) while preserving the sequential generation constraint for inference.
• Contrast with Encoder: Unlike a bidirectional encoder (e.g., in BERT), which uses full attention, the decoder's causal masking is unidirectional, flowing from past to future.

Algorithms that determine how the model selects the next token from its predicted probability distribution. The choice critically balances optimality, diversity, and computational cost.

Key strategies include:

• Greedy Decoding: Selects the token with the highest probability at each step. Fast but can lead to sub-optimal overall sequences.
• Beam Search: Maintains a fixed number (beam width) of most promising partial sequences, often finding higher-probability completions than greedy search.
• Temperature Sampling: Scales logits with a temperature parameter T before applying softmax. T < 1 sharpens the distribution (more deterministic), T > 1 flattens it (more diverse, creative).

A significant challenge where a single incorrect token prediction can derail all subsequent generations, as the model conditions on its own (potentially flawed) outputs. This is known as exposure bias.

• Mitigation Techniques:
  • Scheduled Sampling: During training, occasionally feed the model its own predictions instead of ground truth tokens.
  • Beam Search with Penalties: Apply length or repetition penalties to discourage degenerate sequences.
  • Re-ranking: Generate multiple candidate sequences (e.g., via sampling) and select the best one using a separate scoring model.
• Robotics Impact: A mis-predicted action token can lead to physically invalid or unsafe trajectories, necessitating robust action masking and safety layers.

Autoregressive decoders are almost always conditioned on an external context. In Vision-Language-Action models, this context is provided via cross-attention mechanisms.

• Context Sources:
  • Visual Features: Encoded representations from a perception backbone (e.g., ResNet, ViT).
  • Language Instructions: Embeddings of a natural language command (e.g., 'Open the top drawer').
  • World State: Representations from a world model or current sensor readings.
• Mechanism: The decoder uses its query vectors to attend to key-value pairs from the encoded context, integrating multimodal information to inform each step of action generation.

The Decision Transformer architecture explicitly frames reinforcement learning as an autoregressive sequence modeling problem. It conditions action prediction on a desired return-to-go, past states, and actions.

• Sequence Format: (Return_1, State_1, Action_1, Return_2, State_2, Action_2, ...).
• Autoregressive Process: The model predicts Action_t given the history and the desired future return, treating the return as a kind of 'goal token' that guides the sequence.
• Key Insight: This shifts the paradigm from dynamic programming to conditional sequence generation, leveraging the powerful pattern-matching capabilities of transformers trained via supervised learning on offline datasets.

Causal masking is a technique applied to the self-attention mechanism in a transformer decoder that enforces the autoregressive property by preventing a token position from attending to any future positions in the sequence. This ensures each predicted action token is conditioned only on previously generated tokens, preserving the temporal causality required for valid sequence generation.

• Implementation: A triangular mask of -inf values is added to the attention scores for future positions before the softmax operation.
• Critical Role: It is the architectural guarantee that makes autoregressive decoding possible within parallelizable transformer models.

Beam search is a heuristic search algorithm used during the autoregressive decoding of action sequences to approximate the most likely high-probability output. Instead of greedily selecting the single next-best token, it maintains a fixed-width set (the beam) of the most promising partial sequences, expanding them token-by-token.

• Trade-off: Balances computational cost against output quality, finding better sequences than greedy decoding but at higher memory and compute cost proportional to the beam width.
• Robotics Application: Used in task and motion planning to generate diverse, high-reward action plans from a language-conditioned policy.

Temperature sampling is a technique to control the stochasticity (randomness) of an autoregressive model's output distribution. It scales the logits before applying the softmax function with a temperature parameter T.

• High Temperature (T > 1): Flattens the distribution, increasing diversity and exploration in generated action sequences. Useful for discovering novel behaviors.
• Low Temperature (T < 1): Sharpens the distribution, making the model more deterministic and likely to choose the highest-probability token. Used for precise, repeatable execution.
• Core Mechanism: The modified probability for token i is calculated as P(i) = exp(logit_i / T) / sum(exp(logit_j / T)).

The Decision Transformer is a transformer-based architecture that frames sequential decision-making and action prediction as an autoregressive sequence modeling problem. It takes a trajectory of past states, actions, and a desired future return (reward) as input tokens and generates future actions autoregressively.

• Paradigm Shift: Treats reinforcement learning as a conditional sequence generation task rather than dynamic programming.
• Action Decoding: The model outputs action tokens autoregressively, conditioned on the specified goal (return-to-go) and history, making it a prime example of goal-conditioned autoregressive decoding for control.

Action masking is a constraint technique applied during the autoregressive decoding step to prevent a policy from selecting invalid, unsafe, or physically impossible actions. It works by setting the probability of masked actions to zero before sampling the next token.

• Safety & Feasibility: Enforces kinematic limits, collision avoidance, and task-specific rules (e.g., cannot 'unscrew' before 'grasp').
• Implementation: Often implemented by adding a large negative value (e.g., -1e9) to the logits of invalid actions before the softmax and sampling step in the decoder.

Classifier-free guidance is a technique for strengthening the influence of a conditioning signal (e.g., a language instruction) during the autoregressive sampling process. It works by blending the model's conditional and unconditional predictions.

• Mechanism: The guided logits are computed as guided_logits = (1 + w) * conditional_logits - w * unconditional_logits, where w is a guidance scale.
• Effect: Amplifies the probability of action tokens that are strongly associated with the given instruction, leading to more precise and instruction-following behavior. It is widely used in diffusion policies and can be adapted for autoregressive transformer decoders.