Nucleus Sampling (Top-p)

Unlike top-k sampling, which selects from a fixed number of tokens, nucleus sampling dynamically adjusts the size of the candidate set for each generation step. It calculates the cumulative probability distribution of the model's predicted tokens, sorted in descending order. The algorithm then selects the smallest possible set of tokens where the sum of their probabilities first exceeds the threshold p (e.g., 0.9). This set is called the nucleus. For a sharp distribution, the nucleus may be small; for a flat distribution, it can be large, making the method contextually adaptive.

The core hyperparameter p (top-p) controls the selectivity of the sampling process. It represents a probability mass threshold, typically between 0.7 and 0.95.

• High p (e.g., 0.95): A larger nucleus is selected, including more lower-probability tokens. This increases diversity and creativity in the output but can also raise the risk of incoherence or off-topic text.
• Low p (e.g., 0.7): A smaller, sharper nucleus is selected, restricting choices to only the highest-probability tokens. This produces more conservative, predictable, and grammatically coherent text but can become repetitive. The value is chosen based on the desired trade-off between coherence and novelty for the specific application.

Top-k sampling fixes the number of candidate tokens (k), which can be problematic:

• With a sharp distribution, a high k includes many very low-probability, nonsensical tokens.
• With a flat distribution, a low k may exclude plausible, high-probability candidates. Nucleus sampling solves this by being distribution-aware. It ignores the long "tail" of very low-probability tokens regardless of their rank, focusing computational probability mass on credible next steps. This generally leads to more human-like and contextually appropriate text generation, reducing the incidence of glaring errors that can occur when sampling from a fixed-k tail.

Once the nucleus (set V) is defined, the probabilities of the tokens within it are renormalized to sum to 1. This is done by dividing each token's original probability by the cumulative probability of the nucleus: P'(x) = P(x) / sum(P(V)). Sampling (e.g., multinomial sampling) is then performed over this renormalized distribution. This ensures:

• The model never samples from tokens outside the nucleus.
• The relative likelihood of tokens within the nucleus is preserved.
• The final output is drawn from a focused, high-quality probability distribution tailored to the specific context of the preceding tokens.

A key benefit of nucleus sampling is its automatic and principled truncation of the vocabulary's long tail. In many language model outputs, a vast number of tokens have vanishingly small probabilities. By design, nucleus sampling excludes this entire tail, as its cumulative mass is below the threshold (1 - p). This eliminates the chance of generating highly improbable, often garbled tokens that can break coherence, a common failure mode in pure random sampling or poorly calibrated top-k. The method effectively focuses the model's "creative energy" on the plausible region of the probability distribution.

Nucleus sampling is particularly favored for open-ended generation tasks where diversity and interesting output are valued alongside coherence.

• Creative Writing & Story Generation: A p-value of 0.9-0.95 helps produce varied narrative turns while maintaining plot consistency.
• Dialogue Systems: It enables more engaging and less repetitive conversational responses.
• Code Generation: A lower p (e.g., 0.8) is often used to prioritize syntactically correct and predictable code structures.
• Text Summarization: It helps generate paraphrased summaries that avoid verbatim copying from the source. It is commonly used in conjunction with temperature scaling, which sharpens or flattens the distribution before the nucleus is selected.

Top-k sampling restricts the model's vocabulary to the k most probable next tokens before sampling. Unlike nucleus sampling's dynamic threshold, k is a fixed hyperparameter.

• Mechanism: At each generation step, only the top k tokens by probability are considered. Probabilities are renormalized over this set, and a token is sampled.
• Trade-off: A small k (e.g., 10) makes output more predictable but potentially repetitive. A large k (e.g., 50) increases diversity but can allow low-probability, nonsensical tokens into the candidate pool.
• Key Difference vs. Top-p: Top-k uses a static count; top-p uses a dynamic probability mass.

Temperature is a hyperparameter applied to the logits (pre-softmax scores) of a model to control the randomness of its output distribution before any sampling method (like top-p or top-k) is applied.

• Formula: Adjusted probability = softmax(logits / temperature).
• Effect: A temperature < 1.0 (e.g., 0.7) sharpens the distribution, making high-probability tokens more likely. This yields more focused, deterministic text.
• Effect: A temperature > 1.0 (e.g., 1.5) flattens the distribution, giving lower-probability tokens a better chance. This increases creativity and diversity but also the risk of incoherence.
• Standard practice: Temperature tuning is almost always used in conjunction with top-p or top-k sampling.

Greedy decoding is the simplest decoding strategy where the model selects the single token with the highest probability at every generation step.

• Mechanism: No sampling is involved. The output is argmax(P(token | context)) at each step.
• Result: Produces the locally optimal choice at each step, but often leads to repetitive, generic, and globally sub-optimal text sequences. It lacks exploration.
• Contrast with Sampling: Greedy decoding is deterministic and often fails to capture the natural variation and nuance that probabilistic sampling methods (like top-p) provide. It is rarely used for creative text generation.

Beam search is a deterministic, heuristic search algorithm that explores multiple potential sequence continuations in parallel to find a high-probability overall output.

• Mechanism: It maintains b (beam width) candidate sequences at each step. For each candidate, it considers the top k next tokens, generating b * k possibilities. It then prunes back to the b sequences with the highest cumulative (log) probability.
• Use Case: Predominantly used in tasks where correctness and coherence are paramount and a single 'best' output is desired, such as machine translation or abstractive summarization.
• Trade-off: Can produce more fluent and coherent long-form text than greedy decoding but is computationally heavier and can still suffer from blandness or repetition. It is less common for open-ended dialogue.

Contrastive search is a decoding method designed to reduce repetition by penalizing tokens that are too similar to the recent context, promoting diversity while maintaining coherence.

• Mechanism: At each step, it selects from a set of high-probability candidates (like top-k/p). The final score balances the model's probability with a degeneration penalty. This penalty is based on the cosine similarity between the candidate token's embedding and the embeddings of tokens in the recent context.
• Goal: Suppress tokens that would lead to repetitive loops (e.g., "the the the") or generic continuations, encouraging the model to produce more novel and information-dense text.
• Benefit: Can mitigate common failure modes of standard sampling without requiring model retraining.

Typical sampling is an alternative to nucleus sampling that selects tokens based on how closely their individual information content matches the expected information of the distribution.

• Core Concept: It chooses tokens whose negative log probability (surprisal) is close to the entropy of the distribution. Formally, it samples from the set where | -log(p) - H(p) | < ε.
• Intuition: It filters out tokens that are either too predictable (high probability, low information) or too surprising (low probability, high information). This aims to produce text that is both coherent and consistently informative.
• Result: Empirical studies suggest it can reduce the incidence of both bland, generic outputs and extreme, nonsensical outputs compared to nucleus sampling, offering a different trade-off in the diversity-coherence spectrum.