Temperature Calibration

A complementary decoding strategy that dynamically truncates the vocabulary to the smallest set of tokens whose cumulative probability mass exceeds a threshold p. While temperature reshapes the raw probability distribution, top-p cuts off the long, unreliable tail of low-probability tokens.

• Mechanism: Sorts tokens by probability, selects the minimal set where sum ≥ p (e.g., 0.9).
• Interaction: Low temperature + low top-p = highly deterministic; high temperature + high top-p = maximum diversity.
• Use case: Often combined with temperature to prevent selection of nonsensical tokens even at higher randomness settings.

A truncation method that restricts the model's next-token selection to the K most likely candidates before applying softmax. Unlike top-p, which adapts the candidate pool size based on the shape of the distribution, top-K uses a fixed cutoff regardless of confidence.

• Fixed vocabulary: Only the K highest-probability tokens are considered.
• Rigidity problem: A fixed K can be too restrictive for a peaked distribution or too permissive for a flat one.
• Synergy: Often used as a pre-filter before applying temperature scaling to sharpen or flatten the distribution of the top-K candidates.

A decoding parameter that applies a multiplicative penalty to the logits of tokens that have already appeared in the generated sequence. It directly counteracts the degenerative repetition often observed in language models, especially at higher temperatures.

• Formula: logit_i = logit_i / penalty if token i has been generated previously (typical penalty range: 1.0–1.2).
• Interaction with temperature: High temperature can exacerbate repetition by increasing randomness without semantic grounding; repetition penalty provides a stabilizing counter-force.
• Risk: Excessive penalty values force the model into nonsensical, forced-novelty outputs.

A direct, additive manipulation of the raw logits (pre-softmax scores) for specific tokens before temperature scaling is applied. This allows precise control over the presence or absence of particular words or symbols in the output.

• Mechanism: A positive bias increases a token's probability; a negative bias suppresses it.
• Order of operations: Logit bias is applied before temperature scaling, meaning the temperature parameter will further amplify or dampen the effect of the bias.
• Use case: Forcing the model to never output a specific word, or ensuring a particular formatting token is always considered.

An intrinsic evaluation metric that quantifies how well a probability distribution predicts a sample. In language modeling, it measures the model's uncertainty; a lower perplexity indicates the model is less 'surprised' by the actual next token.

• Relationship to temperature: A temperature of 1.0 yields the model's 'natural' perplexity. Raising temperature increases perplexity (more uncertainty), while lowering it decreases perplexity (more confidence).
• Limitation: Low perplexity does not guarantee factual accuracy or coherence; it only measures statistical fit to the training distribution.
• Formula: The exponentiated average negative log-likelihood of the token sequence.

The simplest decoding strategy that always selects the single token with the highest probability at each step. It is the deterministic extreme of the temperature spectrum, equivalent to setting temperature to 0 (or approaching 0).

• Behavior: Produces the single most statistically likely sequence according to the model.
• Drawback: Highly prone to repetitive loops and dull, generic text because it never explores alternative paths.
• Contrast: The polar opposite of high-temperature sampling; used when absolute determinism and reproducibility are required over creativity.