Top-p (Nucleus) Sampling

Unlike top-k sampling which uses a fixed number of tokens, top-p dynamically adjusts the candidate set size for each generation step. It calculates the cumulative probability distribution of the model's predicted tokens, sorted in descending order. The algorithm then selects the smallest set of tokens where the sum of their probabilities first exceeds the threshold p (e.g., 0.9). This means the candidate pool can be large for uncertain predictions and very small for highly confident ones.

The core parameter p (between 0 and 1) defines the nucleus or cumulative probability mass to sample from. Common values are between 0.7 and 0.95.

• High p (e.g., 0.95): Includes more tokens, leading to more diverse and potentially creative outputs, but with a higher risk of incoherence.
• Low p (e.g., 0.7): Restricts sampling to a smaller set of highly probable tokens, producing more predictable and coherent text, but with less variety. Setting p=1.0 is equivalent to sampling from the full vocabulary, while very low values approach greedy decoding.

Top-p is almost always used in conjunction with the temperature parameter. The standard workflow is:

1. Apply temperature scaling to the logits to control sharpness of the probability distribution.
2. Apply softmax to get the final probabilities.
3. Apply top-p filtering to select the nucleus of viable tokens.
4. Renormalize the probabilities within this nucleus.
5. Sample from this renormalized distribution. This combination allows fine-grained control: temperature adjusts the distribution's entropy, while top-p provides an intelligent, adaptive cutoff.

Top-k sampling selects from the k most likely tokens at every step, regardless of their probability mass.

Key Contrasts:

• Fixed vs. Dynamic Set Size: Top-k is static; top-p adapts to the prediction's certainty.
• Handling of Flat Distributions: In low-confidence scenarios, top-k might include many low-probability tokens, increasing randomness. Top-p excludes long tails of negligible probability.
• Handling of Sharp Distributions: When the model is very confident (one token has ~0.99 probability), top-p will select only that token, acting like greedy decoding. Top-k would still sample from k tokens, adding unnecessary noise. Top-p is generally considered more robust across different tasks.

Top-p is the default decoding strategy in most APIs (e.g., OpenAI's ChatGPT) for open-ended generation tasks because it effectively balances quality and diversity.

Ideal applications include:

• Creative writing and story generation.
• Conversational AI where varied, non-repetitive responses are desired.
• Brainstorming and idea generation. It reduces the likelihood of degenerate, repetitive loops (a common failure mode of pure random sampling) while avoiding the excessive determinism of greedy or beam search.

Not Ideal for Deterministic Tasks: For tasks requiring exact, reproducible outputs (e.g., code generation, data extraction), beam search or greedy decoding with a very low temperature and p-value is preferred.

Parameter Sensitivity: The quality of output can be sensitive to the chosen p value, requiring task-specific tuning.

Computational Overhead: Requires sorting probabilities and computing a cumulative sum each step, adding slight overhead compared to top-k, though this is negligible for most applications.

Does Not Guarantee Factuality: Like all decoding methods, it only controls the selection mechanism from the model's distribution; it does not address underlying model hallucinations or inaccuracies.

Temperature is a hyperparameter that controls the randomness of a model's outputs by scaling the logits (raw prediction scores) before applying the softmax function.

• Low temperature (e.g., 0.1): Makes the probability distribution sharper. The model becomes more deterministic, consistently choosing the most likely tokens. This is ideal for tasks requiring factual accuracy and consistency, like code generation.
• High temperature (e.g., 1.0): Flattens the probability distribution. The model becomes more creative and diverse in its outputs, useful for brainstorming or creative writing.
• Interaction with Top-p: Temperature is applied before top-p sampling. A high temperature can create a flatter distribution, making top-p's selection of the nucleus more diverse.

Top-k sampling is a decoding strategy that restricts the model's next-token selection to the k most probable tokens from its vocabulary.

• Mechanism: At each generation step, only the top k tokens by probability are considered. The probability mass is redistributed among these k tokens (via softmax), and one is sampled.
• Comparison to Top-p: Top-k uses a fixed number (k), while top-p uses a dynamic cumulative probability (p). Top-k can be problematic because the appropriate k can vary wildly depending on the steepness of the probability distribution at each step. Top-p adapts to this variability naturally.
• Use Case: Often used in earlier models (e.g., GPT-2). It's simpler but less adaptive than top-p.

Greedy decoding is the simplest decoding strategy where the model always selects the single most probable token at every generation step.

• Mechanism: No sampling is involved. The algorithm is deterministic: next_token = argmax(P(token | context)).
• Limitations: It often leads to repetitive, generic, and sub-optimal text sequences because it doesn't consider that a slightly less probable token now might lead to a much better overall sequence later (a problem known as search error).
• Contrast: Top-p and other sampling methods introduce controlled randomness to escape these local optima, producing more natural and higher-quality text. Greedy decoding is used when absolute determinism is required.

Beam search is a deterministic decoding algorithm that explores multiple potential sequences in parallel to find a high-probability overall output.

• Mechanism: It maintains b (beam width) most promising sequences at each step. It expands each, keeps the top b sequences by cumulative probability, and repeats. This approximates a broader search for the optimal sequence.
• Use Case: Highly effective for tasks where coherence and likelihood are paramount, such as machine translation or text summarization, where there is often a single "best" answer.
• Contrast with Sampling: Beam search is deterministic and aims to maximize sequence probability. Sampling methods (top-p, top-k) are stochastic and aim for diversity and naturalness, which is often better for open-ended generation like dialogue or story writing.

Logits are the raw, unnormalized output scores generated by a language model's final layer before being converted into a probability distribution.

• Role in Decoding: The model produces a logit for every token in its vocabulary. These values represent the model's relative preference for each token given the context.
• Transformation to Probabilities: The softmax function is applied to the logits to convert them into a proper probability distribution (all values sum to 1).
• Parameter Interaction: Decoding parameters like temperature and top-p operate directly on these logits or the resulting probabilities. Temperature scales the logits before softmax. Top-p is applied to the sorted probabilities after softmax.

Repetition penalty (or frequency/presence penalty) is a technique used to discourage a model from generating repetitive text by artificially reducing the probability of tokens that have already appeared in the recent context or output.

• Mechanism: The logits of tokens that are already in the output sequence are manually decreased by a penalty factor. This makes the model less likely to sample them again immediately.
• Synergy with Top-p: While top-p controls diversity via probability mass, repetition penalty controls diversity via token history. They are often used together. Top-p ensures a dynamic candidate set, while repetition penalty actively discourages re-selection from that set.
• Application: Critical for long-form generation to avoid loops or excessive reuse of phrases. It's a common parameter in inference APIs alongside temperature and top_p.