Top-p (Nucleus) Sampling

Unlike top-k sampling which uses a fixed number of candidates, top-p dynamically adjusts the candidate set size based on the probability distribution shape. When the model is highly confident, the nucleus is small; when uncertain, it expands. This prevents including low-probability tokens in flat distributions while preserving diversity in peaked ones. The algorithm sorts tokens by probability, then cumulatively sums them until the threshold p is reached, discarding all remaining tokens.

The core algorithm operates in three steps:

• Sort: Arrange all tokens in descending order by their predicted probability
• Accumulate: Compute the cumulative sum of probabilities starting from the highest
• Truncate: Retain only tokens whose cumulative sum ≤ p

For example, with p=0.9, if the top token has 0.5 probability and the next has 0.3, the cumulative sum reaches 0.8. The third token at 0.15 pushes it to 0.95, exceeding the threshold, so only the first two tokens form the nucleus.

Top-p sampling exhibits fundamentally different behavior depending on the entropy of the prediction:

• High-confidence context: When one token dominates (e.g., 0.95 probability), the nucleus contains only that single token, effectively behaving as greedy decoding
• Medium-confidence context: A moderate spread of probabilities yields a nucleus of 5-20 tokens, enabling controlled creativity
• Low-confidence context: A flat distribution may require dozens of tokens to reach the p threshold, allowing broader exploration

This adaptivity makes top-p superior to fixed-k approaches for open-ended generation tasks.

Top-p sampling is typically combined with temperature scaling for fine-grained control. Temperature is applied first to reshape the logits distribution, then top-p truncation is applied to the resulting probabilities. A typical production configuration uses:

• Temperature = 0.7 to 0.9 for creative writing
• Top-p = 0.9 to 0.95 for balanced output
• Temperature = 0.2 to 0.5 with top-p = 0.9 for factual, grounded generation

The interplay allows practitioners to independently control distribution sharpness and candidate pool size.

Top-p addresses a fundamental limitation of top-k: fixed candidate counts ignore distribution shape. Consider two scenarios:

• Peaked distribution: Top-k=40 may include tokens with near-zero probability, introducing noise
• Flat distribution: Top-k=40 may exclude viable candidates that collectively represent significant probability mass

Top-p solves both cases by using the probability mass itself as the cutoff criterion. However, many production systems combine both methods, applying top-k truncation first to limit the maximum pool size, then top-p to further refine based on confidence.

Top-p sampling was introduced by Ari Holtzman et al. in 2019 in the paper 'The Curious Case of Neural Text Degeneration' as a solution to the repetitive, degenerate text produced by maximization-based decoding methods like beam search. It demonstrated that sampling from the dynamic nucleus produces text that is both more diverse and more coherent than likelihood-maximizing approaches. The technique has since become a standard default in major frameworks including Hugging Face Transformers, OpenAI's API, and Anthropic's Claude.

A hyperparameter that controls the randomness of predictions by scaling the logits before the softmax function. Unlike top-p, which truncates the distribution, temperature reshapes it.

• T → 0: The distribution sharpens, making the model nearly deterministic (greedy).
• T = 1: The original learned distribution is used.
• T → ∞: The distribution flattens, approaching a uniform random selection.

Temperature is often used in conjunction with top-p to first reshape the probability landscape and then sample from the nucleus.

A predecessor to top-p that restricts sampling to the k most likely next tokens at each step, regardless of the shape of the probability distribution.

• Fixed Cutoff: If k=50, exactly 50 tokens are considered, whether they represent 99% or 10% of the probability mass.
• Rigidity Problem: A flat distribution may need many tokens to cover the nucleus, while a peaked distribution may need very few. Top-k cannot adapt to this dynamic range.

Top-p was designed to solve this rigidity by using a dynamic probability threshold instead of a fixed token count.

A deterministic heuristic that maintains a fixed number (beam width) of the most probable partial sequences at each step to find a near-optimal output.

• Objective: Maximize the overall sequence probability, not just the next token.
• Use Case: Best for tasks with a clear, single correct answer like machine translation or speech recognition.
• Drawback: In open-ended generation, beam search often produces bland, repetitive, and generic text because it strictly follows high-probability paths.

Top-p sampling is preferred for creative tasks where diversity is valued over strict probability maximization.

The simplest decoding strategy: at each step, the model selects the single token with the highest probability.

• Mechanism: Equivalent to temperature sampling with T → 0 or top-k with k=1.
• Result: Purely deterministic output. The same input always produces the same output.
• Failure Mode: Greedy decoding often gets trapped in repetitive loops because it cannot backtrack from a locally optimal but globally poor choice.

Top-p introduces controlled randomness to escape these loops, enabling more natural and varied text generation.

A newer technique that exploits the difference between a large expert model and a smaller amateur model to improve generation quality.

• Mechanism: Rewards tokens that the expert model finds much more likely than the amateur model, filtering out common but uninteresting choices.
• Benefit: Produces more coherent and non-generic text without sacrificing fluency.

Like top-p, contrastive decoding is a method for navigating the probability distribution to avoid degenerate outputs, but it uses a relative signal between two models rather than an absolute probability threshold.

A post-hoc adjustment applied during generation to penalize tokens that have already appeared in the recent context, preventing degenerative loops.

• Mechanism: Scales down the logits of previously generated tokens by a factor (typically 1.0-1.2) before sampling.
• Synergy with Top-p: Top-p truncates the tail of the distribution, but repetition penalty actively reshapes it to discourage redundancy.

These two techniques are frequently combined in production inference pipelines to achieve both diversity (via top-p) and non-repetitiveness (via penalty).