Temperature (LLM Parameter)

Temperature operates on the logits—the raw, unnormalized output scores from the model's final layer—before the softmax function converts them into a probability distribution.

• Formula: The adjusted probability for token i is: P(i) = exp(logit_i / T) / sum(exp(logit_j / T)) where T is the temperature.
• Scaling Effect: A temperature T = 1 applies no scaling. T > 1 flattens the distribution (increasing randomness), while T < 1 sharpens it (increasing determinism).
• This is a direct application of the Boltzmann distribution from statistical mechanics, borrowed to control the 'energy' or randomness of the sampling process.

Low temperature values (e.g., T = 0.2) make the model's output more deterministic and focused.

• Effect: Sharply increases the probability difference between the most likely token and the rest. The model becomes highly confident in its top choices.
• Use Cases: Ideal for tasks requiring high factual accuracy, consistency, and reproducibility.
  • Code generation where syntax must be exact.
  • Summarization or data extraction where deviation is undesirable.
  • Customer service chatbots requiring reliable, on-brand responses.
• Risk: Can lead to repetitive, bland, or overly generic text. May get stuck in loops if the top token is repeatedly selected.

High temperature values (e.g., T = 1.5) make the model's output more creative and diverse.

• Effect: Flattens the probability distribution, giving lower-ranked tokens a much higher chance of being selected.
• Use Cases: Essential for tasks requiring novelty, exploration, and variety.
  • Creative writing and story generation.
  • Brainstorming sessions or idea generation.
  • Chatbots designed for engaging, surprising conversation.
• Risk: Significantly increases the chance of hallucinations, nonsensical outputs, grammatical errors, and incoherence. Outputs become less predictable and reliable.

Temperature is rarely used alone; it is typically combined with top-p (nucleus) sampling for controlled creativity.

• Top-p Sampling: Dynamically selects from the smallest set of tokens whose cumulative probability exceeds p (e.g., 0.9).
• Combined Workflow:
  1. Logits are scaled by temperature.
  2. Softmax converts them to probabilities.
  3. Top-p filtering is applied to the sorted probabilities.
  4. The final token is sampled from this filtered set.
• Synergy: This combination allows developers to set a creativity ceiling (top-p) while using temperature to control the distribution shape within that ceiling. For example, temperature=0.8, top_p=0.95 is a common setting for balanced, coherent, yet non-repetitive dialogue.

Selecting the optimal temperature is an empirical process dependent on the specific model and task.

• Standard Default: T = 1.0 is often a neutral starting point.
• Common Ranges:
  • Highly Deterministic: T = 0.1 to 0.5
  • Balanced / Chat: T = 0.7 to 0.9
  • Creative: T = 1.0 to 1.2
  • Highly Random / Experimental: T > 1.3
• Tuning Process:
  1. Define clear evaluation metrics (e.g., factuality score, diversity score, human preference).
  2. Generate outputs across a temperature sweep (e.g., 0.2, 0.5, 0.8, 1.0, 1.2).
  3. Evaluate outputs quantitatively and qualitatively.
  4. Note that the optimal value is model-dependent; a temperature of 0.8 on GPT-4 does not equate to 0.8 on Llama 3.

Temperature is one of several decoding strategies; it is fundamentally different from top-k sampling.

• Top-k Sampling: A method that restricts sampling to the k tokens with the highest probabilities at each step. It imposes a hard, static limit on the candidate pool.
• Key Distinctions:
  • Scope of Control: Temperature smoothly scales all logits. Top-k is a hard cutoff that completely ignores tokens beyond rank k.
  • Dynamic vs. Static: Top-p (often used with temperature) is dynamic based on cumulative probability. Top-k is static based on rank order.
  • Typical Use: Temperature is more granular for adjusting 'creativity.' Top-k is a blunter instrument to prevent sampling from very low-probability tokens. Top-p has largely superseded top-k in modern practice due to its adaptive nature.

Top-p (nucleus) sampling is a probabilistic decoding strategy that dynamically selects from the smallest set of most probable next tokens whose cumulative probability exceeds a threshold p (e.g., 0.9). Unlike temperature, which scales all logits, top-p prunes the long tail of low-probability tokens first.

• Mechanism: The model's probability distribution is sorted, and tokens are added from most to least probable until the sum of their probabilities > p. Sampling then occurs only from this 'nucleus'.
• Interaction with Temperature: Often used in conjunction with temperature. Temperature is applied first to shape the distribution, then top-p filters it. A common configuration is temperature=0.8, top_p=0.95.
• Benefit: Provides adaptive vocabulary size, preventing erratic low-probability tokens while maintaining diversity for predictable vs. unpredictable contexts.

Top-k sampling is a decoding strategy that restricts the model's next-token selection to the k tokens with the highest probabilities. It is a predecessor to top-p sampling.

• Mechanism: After the model generates logits, only the k most probable tokens are considered; all others are set to zero probability. Sampling then occurs from this truncated distribution.
• Comparison to Top-p: While top-k uses a fixed number of tokens, top-p uses a dynamic probability mass. Top-k can be inefficient—it may include irrelevant tokens if k is too high or exclude all good options if k is too low for a sharp distribution.
• Typical Use: Less common in modern LLM APIs than top-p, but still a foundational concept. It provides a hard limit on output randomness.

Logits are the raw, unnormalized output scores generated by a large language model's final layer for each token in its vocabulary before any final decision is made. Temperature directly operates on these values.

• Definition: A vector of real numbers where each score corresponds to the model's relative preference for each possible next token.
• Transformation: To create a probability distribution, logits undergo softmax: softmax(logits) = exp(logits) / sum(exp(logits)).
• Temperature's Role: The temperature parameter T scales the logits before softmax: softmax(logits / T). A high T (>1) compresses differences between logits, making the distribution more uniform (more random). A low T (<1) exaggerates differences, making the highest logit dominate (more deterministic).

Repetition penalty and frequency penalty are hyperparameters that discourage the model from generating repetitive or common tokens, working alongside temperature to shape output quality.

• Repetition Penalty: Directly penalizes tokens that have already appeared in the recent output. A value >1.0 (e.g., 1.2) reduces the probability of repeated tokens.
• Frequency Penalty: Penalizes tokens based on their overall frequency in the output so far, discouraging common words. It can be positive (penalize) or negative (encourage).
• Key Difference: While temperature controls overall randomness, these penalties provide targeted control over lexical diversity. They are applied to the logits after temperature scaling. Using them with a moderate temperature (e.g., 0.7-0.9) often yields the most coherent and varied text.

Greedy decoding and beam search are deterministic or semi-deterministic decoding strategies that contrast with stochastic methods like temperature sampling.

• Greedy Decoding: Always selects the single token with the highest probability at each step. It is equivalent to temperature sampling with T → 0. It's fast but can lead to repetitive, suboptimal sequences.
• Beam Search: Maintains b (beam width) most probable sequence hypotheses at each step. It explores a limited search space but is still largely deterministic. It often produces fluent but generic text.
• Stochastic vs. Deterministic: For creative tasks (story writing, brainstorming), temperature > 0 is preferred. For factual, closed-ended tasks (translation, summarization), low temperature or beam search is typical. Temperature introduces the randomness needed for diverse, human-like generation.

A seed is a numerical value used to initialize the pseudorandom number generator during LLM sampling. When combined with a fixed temperature, it enables reproducible outputs.

• Function: Setting a specific seed value ensures that the sequence of 'random' choices the model makes during sampling is identical every time the same prompt and parameters are used.
• Interaction with Temperature: The seed controls which random path is taken, while the temperature controls how broad the set of possible paths is. With temperature=0, the seed is irrelevant, as the path is deterministic.
• Critical for Debugging: In production and testing, using a fixed seed and temperature allows engineers to isolate and reproduce model behavior, distinguishing between randomness in the model and changes due to prompt or system modifications.