Temperature

Temperature operates by dividing the raw logit scores by the temperature value before the softmax function is applied. A temperature of T=1.0 leaves the original distribution unchanged. Values approaching T=0 amplify the differences between logits, making high-probability tokens overwhelmingly likely and producing near-deterministic outputs. Values above T=1.0 compress the differences, flattening the distribution and giving lower-probability tokens a greater chance of being selected. This scaling directly controls the sharpness of the probability mass function.

Adjusting temperature is fundamentally an entropy control mechanism. Low temperature (e.g., T=0.2) minimizes variance, making the model consistently choose the most statistically likely token. This is ideal for factual extraction, code generation, and translation where accuracy is paramount. High temperature (e.g., T=1.5) injects noise, enabling the model to escape repetitive loops and produce diverse, unexpected, and creative text. However, excessive temperature leads to incoherence, as the model begins sampling from the long tail of irrelevant tokens.

Temperature is applied before other sampling constraints like Top-P (nucleus sampling) and Top-K. The order of operations is critical:

• Step 1: Logits are divided by the temperature value.
• Step 2: The resulting scaled distribution is filtered by Top-K (keeping only the K highest-probability tokens).
• Step 3: Top-P further truncates the set to the smallest group whose cumulative probability exceeds P. A high temperature flattens the distribution, which can cause Top-P to include a wider variety of tokens, while a low temperature concentrates probability mass into a very small set, often overriding the effect of Top-P.

Effective temperature settings are highly task-dependent and model-specific:

• Structured JSON Generation: Use T=0.0 (greedy decoding) or T=0.1 to guarantee valid syntax and prevent hallucinated keys.
• Conversational Chatbots: A range of T=0.7 to T=0.9 provides a natural, engaging tone without becoming unhinged.
• Creative Storytelling: Values between T=1.0 and T=1.5 encourage novel plot twists and descriptive language.
• Mathematical Proofs: Set to T=0.0 to enforce strict logical step-by-step derivation without random deviation. Note that heavily quantized models may exhibit different sensitivity to temperature changes due to altered weight distributions.

Temperature, Top-P, and Top-K form a triad of distribution control. While temperature reshapes the entire distribution, Top-K imposes a hard cutoff on the number of candidates, and Top-P imposes a dynamic cutoff based on cumulative probability mass. A common misconfiguration is setting high temperature with low Top-P, which creates a contradictory state: the flattened distribution is immediately truncated, often resulting in bland or repetitive text despite the high temperature setting. Best practice is to tune temperature first, then apply Top-P to trim the noisy tail.

Lowering temperature is a primary hallucination suppression technique for enterprise RAG applications. By reducing the sampling variance, the model is forced to rely heavily on the most statistically probable tokens, which in a grounded context are those drawn from the retrieved documents. Setting T=0.0 or T=0.1 ensures the model acts as a near-deterministic summarizer of the provided context rather than an improvisational generator. However, this can also cause the model to get stuck in repetitive loops if the context itself is ambiguous, requiring a balanced approach with structured output constraints.