Temperature Scaling

Temperature (T) is applied within the softmax function to modulate the sharpness of the output distribution. The standard softmax is modified to softmax(z_i / T), where z_i represents the logits or similarity scores.

• Low Temperature (T < 1): Amplifies differences between scores. The highest score dominates the probability mass, creating a sharp, high-confidence distribution. This is akin to a 'greedy' selection.
• High Temperature (T > 1): Smooths the distribution, making probabilities more uniform. This increases the relative weight of lower-scoring items, introducing diversity.
• T = 1: The standard softmax function with no scaling applied.

Temperature scaling is a post-processing method for calibrating model confidence. A well-calibrated model's predicted probability reflects the true likelihood of correctness.

• Overconfidence: A model may output a 99% similarity score but be factually wrong. A low temperature exacerbates this.
• Calibration: By finding an optimal T on a validation set, you can align predicted probabilities with empirical accuracy without changing the underlying ranking order.
• Effect on Vector Search: In a RAG pipeline, a calibrated temperature ensures that retrieved documents with a 0.9 probability are genuinely more relevant than those at 0.6, improving downstream factual grounding.

In knowledge distillation, temperature scaling is used to extract 'dark knowledge' from a large teacher model to train a smaller student model.

• Soft Targets: A high temperature (e.g., T=5) is applied to the teacher's logits to produce a softer probability distribution over classes.
• Rich Information: This soft distribution reveals inter-class similarities learned by the teacher (e.g., a 'cat' image might have a small probability for 'dog'), providing richer gradients than hard labels.
• Training Process: The student is trained with the same high temperature, and T is reset to 1 for inference. This technique is foundational for compressing large embedding models into smaller, faster ones for vector search.

In autoregressive generation, temperature directly controls the creativity vs. determinism of the output by scaling logits before sampling.

• T → 0: Approximates greedy decoding. The model always selects the token with the highest probability, leading to deterministic, often repetitive text.
• T = 0.7: A common default for balanced creativity. It dampens the probability of low-likelihood tokens while still allowing for variation.
• T > 1.5: The distribution flattens significantly, causing the model to sample rare, often nonsensical tokens, leading to hallucinations and incoherent text. This is distinct from Top-k or Top-p sampling, which truncate the distribution.

Temperature is a crucial hyperparameter in contrastive loss functions like InfoNCE, which are used to train embedding models for semantic similarity.

• Hard Negative Mining: A low temperature focuses the loss heavily on hard negatives (dissimilar items that are close in the embedding space), forcing the model to learn fine-grained distinctions.
• Uniformity vs. Tolerance: A high temperature treats many negatives as equally bad, promoting a more uniform distribution of embeddings on the hypersphere but potentially tolerating semantic overlap.
• Training Dynamics: The choice of T balances the concentration of the loss on the hardest negatives against the overall uniformity of the representation space, directly affecting the quality of the resulting vector space positioning.

Temperature scaling can be combined with logit adjustment to correct for imbalanced training data distributions without retraining the model.

• The Problem: Models trained on imbalanced data develop a bias toward majority classes, skewing the embedding space.
• Post-hoc Correction: By subtracting the log of the class prior from the logits before applying temperature scaling, you can shift the decision boundary to favor rare classes.
• Formula: adjusted_logit = logit - log(prior). Applying temperature after this adjustment allows you to fine-tune the confidence of these corrected predictions, which is essential for maintaining retrieval accuracy for niche, long-tail concepts in a vector database.