Attention Temperature

Temperature (T) is applied as a divisor to the logits before the softmax function: softmax(logits / T). This mathematical operation directly controls the variance of the unnormalized attention scores.

• T = 1.0: Standard softmax, preserving the original logit distribution.
• T → 0: Approximates an argmax, concentrating all probability mass on the single highest logit.
• T → ∞: Flattens the distribution toward a uniform distribution, giving all tokens equal attention weight.

The scaling factor effectively sharpens or smooths the probability landscape without altering the relative ordering of token importance.

Temperature directly governs the entropy of the attention weight distribution, providing a tunable knob for context selectivity.

• Low Temperature (T < 1): Produces a peaky distribution. The model attends almost exclusively to the single most relevant token, enforcing strict focus. This is beneficial for tasks requiring precise copying or factual extraction.
• High Temperature (T > 1): Produces a diffuse distribution. Attention is spread across many tokens, allowing the model to aggregate information from a broader context. This aids in summarization and understanding general themes.

This mechanism allows a single attention head to dynamically shift between a precise lookup and a global averaging operation.

The role of temperature diverges significantly between the training and inference phases of a Transformer model.

• During Training: Temperature is typically fixed at 1.0. The model learns to produce appropriately scaled logits through backpropagation, making an explicit temperature parameter redundant for standard pre-training.
• During Inference: Temperature becomes a critical post-training control for generative models. It is used to modulate the creativity-determinism trade-off in next-token prediction, often in conjunction with top-k and top-p sampling.

This separation means temperature is primarily a behavioral tuning parameter applied after the model's weights are frozen.

Temperature plays a foundational role in knowledge distillation, where a smaller student model is trained to replicate the output distribution of a larger teacher model.

• A high temperature (e.g., T > 5) is applied to the teacher's logits to soften its probability distribution.
• This softened distribution reveals the inter-class similarities learned by the teacher, providing richer supervisory signal than hard labels alone.
• The student is trained with the same high temperature, and the loss function compares the softened distributions.

This technique transfers the dark knowledge of the teacher's generalization patterns, enabling effective model compression.

Temperature scaling is the simplest and most effective method for post-hoc calibration of neural network confidence scores.

• Modern deep networks are often overconfident, producing probability estimates that do not reflect the true likelihood of correctness.
• A single scalar temperature parameter is optimized on a held-out validation set to minimize the Expected Calibration Error (ECE).
• This process does not alter the model's accuracy or ranking of predictions; it only rescales the logits to make the output probabilities more reliable.

Proper calibration is essential for risk-sensitive applications like medical diagnosis and autonomous driving where probability estimates inform critical decisions.

Temperature operates on the continuous logit space before discretization, distinguishing it from truncation-based sampling methods.

• Temperature: Reshapes the entire probability distribution smoothly. All tokens remain eligible, but their relative probabilities are adjusted.
• Top-k Sampling: A hard truncation that eliminates all but the k most likely tokens, setting the rest to zero probability.
• Top-p (Nucleus) Sampling: A dynamic truncation that retains the smallest set of tokens whose cumulative probability exceeds a threshold p.

These methods are often combined. A typical generative pipeline applies temperature scaling first to control sharpness, then applies top-p sampling to eliminate the long tail of improbable tokens.

The fundamental mathematical operation that attention temperature directly modifies. It computes the dot product of Query and Key vectors, scales the result by 1/√dₖ, and applies a softmax function to obtain attention weights.

• The temperature parameter is applied to the logits before this softmax normalization
• A higher temperature makes the softmax distribution more uniform
• A lower temperature makes the softmax distribution peakier, approaching a hard maximum

A generalization of the standard softmax function where logits are divided by a temperature parameter T. When T > 1, the output distribution flattens, increasing entropy and diversity. When T < 1, the distribution sharpens, concentrating probability mass on the highest logit.

• T → 0: Approaches argmax (deterministic)
• T = 1: Standard softmax
• T → ∞: Approaches uniform distribution
• Commonly used in knowledge distillation to soften teacher model outputs

An N × N matrix where each entry represents the raw compatibility score between a Query token and a Key token. Temperature scaling is applied to this entire matrix before softmax normalization along the Key dimension.

• The matrix encodes pairwise token relationships across the full sequence
• Temperature directly controls the entropy of each row in the normalized attention weight matrix
• Critical for managing the concentration vs. dispersion of attention across context

The raw, unnormalized scores output by a model before any activation function is applied. In the attention mechanism, logits are the dot products of Query and Key vectors (optionally scaled by √dₖ).

• Temperature divides these logits before the softmax operation
• High-magnitude logits dominate after softmax unless temperature is increased
• Logit scaling is a form of confidence calibration for attention distributions

A module that runs multiple attention operations in parallel, each with its own learned Query, Key, and Value projections. Temperature can be applied per-head or globally across all heads.

• Different heads may benefit from different temperature values
• Some implementations use learnable per-head temperatures
• Allows the model to simultaneously focus sharply on one subspace while exploring broadly in another

A measure of uncertainty in the normalized attention weights. High entropy means attention is spread across many tokens; low entropy means it is concentrated on few.

• Temperature is the primary mechanism for controlling attention entropy
• Low entropy (low T) risks overfitting to spurious patterns
• High entropy (high T) can dilute relevant signal
• Entropy monitoring is used in anomaly detection for attention patterns