Top-P Sampling

Unlike Top-K sampling, which uses a fixed number of tokens, Top-P dynamically adjusts the candidate set size based on the shape of the probability distribution. When the model is highly confident, the pool shrinks to a single token; when uncertain, it expands to include more diverse options. This prevents the inclusion of low-probability 'tail' tokens in flat distributions while preserving diversity in sharp distributions.

The algorithm sorts all tokens by descending probability, then iteratively adds them to the candidate set until the sum exceeds the hyperparameter P (typically 0.9). Only tokens within this 'nucleus' are sampled. For example, with P=0.9 and token probabilities [0.5, 0.3, 0.15, 0.04, 0.01], only the first three tokens are considered because 0.5 + 0.3 + 0.15 = 0.95 ≥ 0.9.

Temperature is applied before Top-P filtering. Temperature scales the logits, sharpening (T < 1) or flattening (T > 1) the probability distribution. A low temperature concentrates mass on the top tokens, making the nucleus smaller; a high temperature spreads mass across more tokens, expanding the nucleus. The two parameters work in concert: temperature shapes the distribution, Top-P truncates it.

Top-K selects a fixed number K of the highest-probability tokens regardless of context. This creates a brittle trade-off:

• K too small: Repetitive, dull output in flat distributions
• K too large: Inclusion of nonsensical tokens in sharp distributions

Top-P solves this by adapting to the entropy of the prediction. In practice, many implementations combine both: first applying Top-K (e.g., K=50) as a safety ceiling, then Top-P to dynamically narrow the set.

Top-P is a standard parameter across inference engines:

• Hugging Face Transformers: do_sample=True, top_p=0.95
• OpenAI API: top_p parameter (default 1.0, meaning disabled)
• vLLM: SamplingParams(top_p=0.9)
• llama.cpp: --top-p 0.9 flag

Most frameworks default to P=0.9 or P=0.95, striking a balance between coherence and creativity.

Tuning P controls the creativity-coherence spectrum:

• P=0.1-0.5: Highly deterministic, suitable for factual Q&A and code generation
• P=0.9: Standard creative writing and conversation
• P=0.95-1.0: Maximum diversity for brainstorming and ideation

A common production pattern is to set temperature=0.7, top_p=0.9 for general-purpose chat, and temperature=0.2, top_p=0.1 for structured extraction tasks.

A hyperparameter that controls the randomness of predictions by scaling the logits before softmax. Higher values (e.g., 1.0) flatten the distribution, making less likely tokens more probable. Lower values (e.g., 0.1) sharpen it, making the model more deterministic.

• Mechanism: Divides logits by the temperature value before softmax
• Range: Typically 0.0 to 2.0
• Interaction: Often combined with Top-P; temperature is applied first to reshape the distribution, then nucleus sampling truncates it

A predecessor to Top-P that restricts the candidate pool to the K most probable tokens at each step, then samples from that fixed-size subset. Unlike Top-P, the candidate count remains constant regardless of the distribution shape.

• Limitation: A fixed K may include low-probability tokens in a flat distribution or exclude reasonable options in a peaked one
• Advantage: Simpler to implement and reason about
• Common values: K=40 or K=50 in many implementations

The simplest decoding strategy that always selects the single token with the highest probability at each step. This is equivalent to Top-P with P=0 or temperature=0.

• Behavior: Fully deterministic; identical prompts always produce identical outputs
• Use case: Factual tasks requiring reproducibility, such as code generation or structured data extraction
• Drawback: Can produce repetitive, dull, or degenerate text in creative contexts

A deterministic search algorithm that maintains multiple candidate sequences (beams) in parallel and selects the one with the highest cumulative probability. Unlike sampling methods, it optimizes for the globally most likely sequence.

• Beam width: Number of parallel hypotheses tracked (typically 3-10)
• Trade-off: Higher quality for translation and summarization, but can produce bland, generic outputs
• Contrast: Top-P introduces controlled randomness; beam search is purely deterministic

A decoding parameter that penalizes tokens already generated in the current sequence to discourage repetitive loops. Applied by reducing the logits of previously seen tokens before sampling.

• Mechanism: Multiplies logits by a penalty factor (typically 1.0-1.3) for tokens that have appeared
• Synergy: Works alongside Top-P to maintain diversity within the nucleus candidate set
• Risk: Excessive values degrade coherence by forcing unnatural word choices

A technique that constrains model output to conform to a predefined formal grammar or schema (e.g., JSON, regex). It overrides sampling by masking invalid tokens at each step.

• Implementation: Token masks are applied post-softmax to zero out grammar-violating candidates
• Relationship: Top-P still operates on the remaining valid tokens to introduce diversity within constraints
• Use case: API responses, data extraction, and any task requiring parseable output