Temperature Sampling

Temperature operates by dividing the raw logits by a scalar value T before the softmax function is applied. This mathematical operation reshapes the probability distribution without altering the relative ranking of tokens.

• T → 0: Logits are amplified, making the highest-probability token overwhelmingly likely. The model becomes deterministic.
• T = 1: The original softmax distribution is preserved unchanged.
• T → ∞: Logits are compressed toward uniformity, making all tokens nearly equally probable. Output becomes random noise.

The scaling effect is non-linear: a temperature of 0.7 does not simply reduce randomness by 30%—it exponentially sharpens the peak of the distribution.

Temperature directly governs the entropy of the sampling process, which is the measure of uncertainty in the next-token selection. Higher entropy produces more surprising, less predictable outputs.

• Low temperature (0.1–0.5): Produces focused, repetitive, and highly coherent text. Ideal for factual extraction, code generation, and structured data tasks.
• Medium temperature (0.7–0.9): Balances coherence with variety. Suitable for general-purpose conversation and content drafting.
• High temperature (1.0–1.5): Introduces significant lexical diversity and conceptual leaps. Useful for creative writing, ideation, and exploring alternative phrasings.

At extreme values above 2.0, the model's outputs degrade into nonsensical token sequences as the probability distribution flattens completely.

Temperature is rarely used in isolation. It is typically combined with top-k and top-p (nucleus) sampling to create a multi-layered filtering pipeline.

• Temperature + Top-k: Temperature first reshapes the distribution, then top-k truncates it to the k most probable tokens. This prevents the long tail of low-probability tokens from being selected even at higher temperatures.
• Temperature + Top-p: Temperature scales the distribution, then nucleus sampling dynamically selects the smallest set of tokens whose cumulative probability exceeds p. This adapts the candidate pool size to the shape of the distribution.
• Order of operations matters: Temperature is always applied before truncation. Applying it after would defeat the purpose of reshaping the distribution.

A common production configuration is T=0.8 with top-p=0.95, which provides controlled diversity without risking incoherence.

Optimal temperature values are highly task-dependent and should be calibrated against evaluation metrics rather than set arbitrarily.

• Code generation: T=0.0–0.3. Syntactic correctness demands near-deterministic outputs.
• Factual question answering: T=0.0–0.2. Hallucination risk increases sharply with temperature.
• Summarization: T=0.3–0.7. Some flexibility improves readability without sacrificing fidelity.
• Creative writing: T=0.8–1.2. Higher values produce stylistic variation and unexpected metaphors.
• Brainstorming and ideation: T=1.0–1.5. Maximum diversity is desired to explore the solution space.

These ranges are starting points. Systematic A/B testing with human evaluation or automated metrics like BLEU and ROUGE should determine the final value.

Temperature and top-p are often confused, but they control randomness through fundamentally different mechanisms.

• Temperature reshapes the entire probability distribution by scaling logits. It changes the relative probabilities of all tokens simultaneously.
• Top-p performs a hard cutoff, eliminating low-probability tokens from consideration entirely regardless of their reshaped probabilities.

A critical distinction: temperature can make an improbable token more likely, but it can never eliminate a token from consideration. Top-p can completely exclude tokens. This means temperature alone cannot prevent the model from selecting a nonsensical token if that token had a non-zero probability in the original distribution.

For production systems requiring both creativity and safety, the combination of moderate temperature with a conservative top-p value provides the most robust control surface.

Setting temperature to zero is the standard method for achieving deterministic outputs from language models, but this behavior depends on the inference implementation.

• T=0 with greedy decoding: The model always selects the single highest-probability token. Output is fully deterministic given the same input and model weights.
• T=0 with beam search: Multiple candidate sequences are maintained, but the final selection is still deterministic.
• Floating-point caveat: In distributed inference across multiple GPUs, minor numerical precision differences can cause non-determinism even at T=0. Reproducibility requires identical hardware and software configurations.
• Seed control: Some inference engines require explicit random seed setting even at T=0 to guarantee bit-for-bit identical outputs across runs.

For enterprise applications requiring auditable, repeatable outputs—such as legal document generation or regulatory filings—T=0 with verified deterministic inference is essential.

The raw, unnormalized scores output by the final layer of a neural network before conversion to probabilities. Temperature sampling operates directly on these logits by dividing them by the temperature constant prior to the softmax function. The softmax then exponentiates and normalizes the scaled logits into a probability distribution. A temperature of 0 would theoretically cause division by zero, so in practice it triggers greedy decoding instead. Understanding this pipeline is essential for implementing custom sampling logic.

The deterministic baseline against which temperature sampling is measured. At each step, the model simply selects the single token with the highest probability. This is equivalent to setting temperature to 0. While perfectly reproducible, greedy decoding often produces degenerate, repetitive text and gets trapped in local maxima. It is suitable for tasks requiring strict factual extraction but fails catastrophically on open-ended generation where diversity is required.

A precursor to top-p sampling that restricts the model's choice to the k most likely next tokens. This is a static truncation method, unlike the dynamic threshold of nucleus sampling. The primary weakness is that a fixed k is blind to the shape of the distribution: for a peaked distribution, k may include unreliable tokens, while for a flat distribution, k may exclude valid options. Top-k is often combined with temperature to first sharpen or flatten the distribution before the static cutoff is applied.

A post-hoc adjustment often used alongside temperature to combat degenerative looping. This parameter applies a multiplicative penalty to the logits of tokens that have already appeared in the context window. By reducing the probability of recently generated tokens, it forces the model to explore new vocabulary. A penalty of 1.0 applies no effect, while values like 1.2 significantly discourage repetition. Over-penalization leads to nonsensical output as common function words become forbidden.

The broader category of techniques aimed at reducing factual fabrication, of which temperature tuning is a critical component. Lowering the temperature reduces the model's freedom to 'invent' statistically plausible but factually incorrect tokens. However, this is a blunt instrument. Advanced mitigation combines low-temperature sampling with grounding attribution and constrained decoding to force outputs to align with a verified knowledge base. Temperature alone cannot guarantee factuality; it only controls statistical variance.