Temperature Sampling

The temperature parameter (τ) is a scalar value applied to the logits (pre-softmax scores) of a model's output distribution. It directly controls the entropy of the resulting probability distribution:

• τ → 0: Logits are scaled to extreme values. The softmax output approaches a one-hot vector, making the model deterministic (always selects the highest-probability token).
• τ = 1: No scaling is applied. The model uses its native, calibrated confidence.
• τ > 1: Logits are compressed toward zero. The softmax output becomes more uniform, increasing diversity and randomness in the sampled output.

The core mathematical operation is the scaling of model logits z before applying the softmax function to obtain the final probability distribution for sampling: P(token_i) = softmax(z_i / τ)

• This scaling is a post-processing step applied during inference, after the model's forward pass.
• It does not require retraining the model.
• The operation is computationally trivial but has a profound effect on the sampling policy, influencing whether a robot's action sequence is cautious and predictable or exploratory and creative.

Adjusting the temperature creates a fundamental trade-off critical for action generation:

• Low Temperature (τ < 1): Produces high-quality, predictable actions. Ideal for precise, repeatable tasks like assembly or reaching a specific coordinate. Risk: Can lead to mode collapse, where the model gets stuck generating repetitive or overly safe action sequences.
• High Temperature (τ > 1): Encourages exploration and diversity. Useful for discovering novel strategies, recovering from errors, or tasks requiring creativity. Risk: Can generate nonsensical or unsafe actions (e.g., erratic joint movements) as low-probability tokens are selected more often.

Temperature sampling is often combined with other sequence generation techniques:

• Top-k / Top-p (Nucleus) Sampling: Temperature is applied first to the logits. Then, Top-k or Top-p filtering is performed on the resulting distribution. This combination allows control over both randomness (temperature) and the set of considered tokens (filtering).
• Beam Search: While beam search traditionally uses deterministic argmax, a temperature-scaled softmax can be used to calculate the probabilities for beam scoring, introducing soft stochasticity into the search.
• Greedy Decoding: Greedy decoding (always choosing argmax) is equivalent to using a temperature of τ → 0.

In Vision-Language-Action models, temperature critically affects physical behavior:

• Task Execution (Low τ): For precise 'pick cup from table' commands, low temperature ensures the model consistently generates the correct, safe grasp and trajectory tokens.
• Exploration & Learning (High τ): During reinforcement learning or when a robot is 'searching for a tool in a cluttered drawer', higher temperature can help it try varied pushing and grasping motions.
• Safety via Action Masking: Even with high temperature, action masking must be used to set probabilities of physically impossible or dangerous actions to zero, ensuring exploration stays within safe bounds.

Selecting the optimal temperature is an empirical, task-dependent process:

• Start with τ = 1.0 to evaluate the model's baseline behavior.
• Iteratively adjust in small increments (e.g., 0.2 to 0.8 for more determinism, 1.2 to 1.8 for more diversity).
• Evaluate quantitatively using task success rate and qualitatively by observing action sequence smoothness and goal achievement.
• Dynamic Temperature: Advanced systems may use annealed temperature schedules (starting high for exploration, ending low for precise execution) or condition the temperature on task complexity or uncertainty estimates.

A differentiable approximation to sampling from a categorical distribution, enabling gradient-based optimization through discrete token selections. It is a key technique for training models that perform action tokenization, as it allows backpropagation through the inherently non-differentiable process of selecting a discrete token from a codebook.

• Mechanism: Uses the Gumbel-Max trick to draw a sample, then approximates the non-differentiable argmax with a softmax function using a temperature parameter.
• Application: Crucial for training VQ-VAE-based tokenizers where the model must learn a discrete latent codebook for continuous actions.
• Relation to Temperature: Its temperature parameter controls the 'sharpness' of the approximation, similar to its role in standard softmax sampling.

A method for backpropagating gradients through discrete, non-differentiable operations (like argmax or token selection) by using the gradient of a continuous surrogate (like softmax) during the backward pass. It is a simpler, often more stable alternative to Gumbel-Softmax for certain tasks.

• Forward Pass: Uses the discrete, hard selection (e.g., the index of the chosen token).
• Backward Pass: Treats the non-differentiable operation as if it were the identity function, passing gradients through unchanged from the continuous surrogate.
• Use Case: Commonly employed in conjunction with vector quantization layers in models like VQ-VAE to train the codebook and encoder/decoder networks.

A heuristic search algorithm used in autoregressive decoding to find a high-probability sequence of tokens (like an action plan). It explores multiple potential futures in parallel, maintaining a fixed number (beam width) of the most promising partial sequences at each step.

• Contrast with Sampling: Unlike temperature sampling, which introduces randomness, beam search is a deterministic (for a given beam width) search for the most likely sequence.
• Application in Robotics: Used to generate coherent, multi-step action sequences from a policy or decision transformer, ensuring temporal consistency.
• Limitation: Can lead to overly generic or repetitive outputs, as it prioritizes high-probability tokens and may miss diverse, high-reward alternatives.

A technique for amplifying the influence of a conditioning signal (like a language instruction or goal image) in conditional generative models, including diffusion policies and some autoregressive models. It works by blending the model's conditional and unconditional predictions during sampling.

• Mechanism: The final output is guided by: output = unconditional_output + guidance_scale * (conditional_output - unconditional_output).
• Effect: A higher guidance scale makes the output more tightly adhere to the condition, reducing diversity but increasing precision—a trade-off analogous to lowering temperature.
• Robotics Use: Critical in diffusion policies and goal-conditioned models to ensure generated actions reliably achieve the specified task, similar to using low temperature for deterministic control.

A technique that prevents a policy from selecting invalid or unsafe actions by setting their probabilities to zero during the action selection step. It is a form of hard constraint enforcement distinct from the soft probabilistic shaping of temperature sampling.

• Implementation: Applied after the model's final softmax layer, before sampling or argmax.
• Purpose: Enforces physical constraints (e.g., joint limits), safety rules (e.g., no collisions), or task logic (e.g., can't place an object before grasping it).
• Relation to Sampling: Temperature scaling is applied before masking; it controls the randomness of the valid action distribution. Masking then removes invalid options entirely.

The sequential generation process where a model predicts the next token in a sequence conditioned on all previously generated tokens. This is the fundamental decoding scheme used by transformer-based action generators where temperature sampling is applied.

• Process: At each timestep t, the model consumes tokens 1...t-1 to produce a probability distribution for token t. Temperature sampling (or another method like argmax) selects the token for this step.
• In Robotics: Used by Decision Transformers and other sequence models to generate action token a_t based on past states, actions, and a goal or return-to-go signal.
• Cumulative Effect: The randomness introduced by temperature at each step compounds, making the overall sequence more or less diverse.