Contrastive Decoding

The core mechanism uses two models: a large expert model (e.g., GPT-3 175B) and a smaller amateur model (e.g., GPT-2 1.5B). At each decoding step, the log probabilities of the amateur are subtracted from the expert's log probabilities. Tokens that the expert finds disproportionately likely compared to the amateur are promoted. This suppresses generic, high-probability filler words that both models agree on, surfacing more informative and contextually specific vocabulary.

A critical filtering step prevents the selection of nonsensical tokens. The algorithm applies a plausibility constraint using a threshold (alpha) on the expert model's probability distribution. Any token whose probability under the expert model falls below this threshold is masked out entirely. This ensures that the contrastive search only considers tokens that the expert model deems reasonable, preventing the amateur model's low-probability artifacts from dominating the selection.

Contrastive decoding inherently penalizes repetitive loops. Degenerate sequences where a model repeats the same phrase arise from a self-amplification feedback loop where high-probability tokens keep getting selected. Because the amateur model shares this tendency toward repetition, the contrastive subtraction cancels out these repetitive signals. The expert's preference for novel, context-appropriate tokens is isolated, leading to more diverse and natural long-form generation.

Smaller amateur models are more prone to hallucination and generic confabulation. By contrasting against this weaker distribution, the decoding process filters out tokens associated with low-confidence, unsupported statements. The expert model's internalized factual knowledge is amplified. Research shows contrastive decoding improves performance on closed-book QA and factuality benchmarks without requiring external retrieval augmentation.

The technique requires running two forward passes per token: one for the expert and one for the amateur. This doubles the inference compute compared to standard greedy decoding. However, the amateur model is typically 10-50x smaller, making the additional cost marginal. For API-based deployments where only the expert model is accessible, proxy amateur models (e.g., a smaller open-source model) can be used, though this introduces distribution mismatch risks.

Decoding by Contrasting Layers (DoLa) is a variant that eliminates the need for a separate amateur model. It contrasts the logit outputs from a later, mature transformer layer against an earlier, premature layer within the same model. Early layers capture low-level linguistic patterns, while later layers encode factual knowledge. The contrast amplifies factual signals without external models, making it a lightweight alternative for reducing hallucinations in single-model deployments.

A layer-contrastive method that amplifies factual knowledge by subtracting the logit output of an early, premature transformer layer from a later, mature layer. Unlike contrastive decoding, DoLa requires no external amateur model—it contrasts internal model states. The difference highlights tokens where the mature layer's knowledge diverges from the early layer's linguistic priors, effectively surfacing factual content while suppressing hallucinations.

A direct manipulation technique that adds a constant bias value to specific tokens' raw prediction scores (logits) before the softmax sampling step. Developers can forcefully increase or decrease the probability of certain words appearing. For example, a positive bias on \n encourages paragraph breaks, while a negative bias on repetitive tokens reduces redundancy. This provides deterministic, hard control over output vocabulary without modifying model weights.

A decoding constraint that prevents the model from generating any sequence of n consecutive tokens that has already appeared in the context window. This eliminates repetitive phrasing at a granular level without requiring an external model or complex probability manipulation. Common implementations block trigrams (n=3) or 4-grams. Unlike contrastive decoding, which addresses overall quality, n-gram blocking specifically targets lexical repetition and is often combined with other decoding strategies.

An algorithm that selects tokens or passages by balancing two competing objectives: relevance to the query and novelty relative to already-selected content. MMR applies a diversity constraint (lambda parameter) that penalizes candidates too similar to existing selections. While contrastive decoding amplifies desirable model behaviors, MMR operates at the selection layer to produce comprehensive, non-redundant outputs—critical for multi-document summarization and information retrieval.

A controlled generation method that forces a language model to output text conforming to a predefined formal grammar (e.g., JSON schema, regular expressions). During decoding, tokens that would violate the grammar are masked from sampling. This guarantees syntactically valid structured output without post-processing. Unlike contrastive decoding's focus on quality amplification, grammar constraints ensure structural compliance—essential for API responses, data extraction, and tool calling.

A decoding parameter that penalizes the selection of tokens or passages that are semantically similar to content already generated. Implemented through repetition penalties, frequency penalties, or presence penalties in sampling algorithms. A higher diversity constraint produces more varied vocabulary and concepts, while lower values permit repetition. This operates at the sampling strategy level, complementing contrastive decoding's probability-space manipulation to produce both high-quality and varied outputs.