N-gram Blocking

During autoregressive generation, the model maintains a lookup table of all n-grams present in the preceding context. Before sampling the next token, the algorithm identifies any candidate token that would complete a previously observed n-gram and sets its logit to negative infinity, effectively assigning it a probability of zero. This forces the model to select alternative, non-repetitive phrasing.

• Operates at the logit manipulation stage before softmax sampling
• Blocking is applied to the exact n-gram, not semantic similarity
• Commonly implemented with n=4 for phrase-level repetition control

The choice of n directly controls the granularity of repetition prevention. Smaller values like n=2 or n=3 block common bigrams and trigrams, which can overly constrain natural language and produce stilted, unnatural output. Larger values like n=5 or n=6 only block longer, more distinctive phrases, allowing natural lexical repetition while preventing verbatim phrase duplication.

• n=1: Blocks individual token reuse (too aggressive, breaks grammar)
• n=3: Blocks common trigrams (moderate constraint)
• n=4: Standard default for phrase-level blocking
• n=6+: Only blocks highly distinctive multi-word sequences

N-gram blocking is a hard constraint that absolutely prohibits exact n-gram recurrence, while a repetition penalty applies a soft, continuous penalty to tokens based on their prior frequency. The penalty approach reduces probability without fully eliminating it, allowing repetition if the model is sufficiently confident.

• N-gram Blocking: Binary, deterministic, zero probability for blocked sequences
• Repetition Penalty: Continuous, probabilistic, scales with prior occurrence count
• Blocking prevents exact matches; penalty discourages but permits them
• Both can be combined for layered repetition control

N-gram blocking is particularly valuable in abstractive summarization where models tend to copy phrases verbatim from the source or repeat their own generated phrases. By blocking n-grams already present in the input document or the partially generated summary, the model is forced to paraphrase and synthesize rather than extract.

• Prevents copy-paste summarization from source documents
• Encourages genuine abstraction and rephrasing
• Reduces self-repetition loops in long-form generation
• Works synergistically with diversity constraints like MMR

N-gram blocking functions as one component within a broader controlled generation pipeline. It operates alongside other logit manipulation techniques such as logit bias, grammar-constrained decoding, and contrastive decoding. The blocking logic is typically applied as a post-processing step on the raw logits before the final softmax normalization.

• Applied after temperature scaling but before sampling
• Compatible with beam search and nucleus (top-p) sampling
• Can be implemented via Hugging Face's NoRepeatNGramLogitsProcessor
• Often paired with diversity constraints for multi-document summarization

While effective at eliminating verbatim repetition, n-gram blocking introduces specific failure modes. It cannot distinguish between unwanted repetition and legitimate recurrence of domain-specific terminology, proper nouns, or fixed phrases. Overly aggressive blocking can degrade factual consistency by forcing the model to use imprecise synonyms for technical terms.

• May replace precise terminology with vague alternatives
• Cannot detect semantic repetition (same meaning, different words)
• Increases computational overhead for large context windows
• Does not address hallucination or factual errors

A generation technique that improves text quality by searching for tokens that maximize the probability difference between a strong expert model and a weaker amateur model. By amplifying behaviors where the expert is confident and the amateur is not, it suppresses repetitive, dull, or generic outputs. This approach requires no additional training and works at inference time by contrasting logits from two models of different sizes or capabilities.

A parameter in decoding algorithms that penalizes repetition and encourages the selection of semantically varied tokens. Unlike n-gram blocking's hard prohibition on exact matches, diversity constraints apply a soft penalty that reduces the probability of tokens similar to those already generated. This produces more lexically varied output while still allowing natural repetition when contextually appropriate.

A widely used decoding parameter that applies a multiplicative penalty to the logits of tokens that have already appeared in the generated sequence. Common implementations include:

• Repetition penalty (1.0–1.5): Scales down scores of previously generated tokens
• Frequency penalty: Penalizes tokens proportionally to how often they've appeared
• Presence penalty: Applies a flat penalty regardless of frequency

Unlike n-gram blocking, this allows repetition if the model is sufficiently confident.

A parameter that directly modifies the raw prediction scores (logits) of specific tokens before sampling. Developers can forcefully increase or decrease the probability of certain words appearing. While n-gram blocking prevents multi-token sequences, logit bias operates at the individual token level and can be used to:

• Suppress specific words entirely
• Boost preferred terminology
• Enforce brand-safe vocabulary

A controlled generation method that forces a language model to output text strictly conforming to a predefined formal grammar, such as a JSON schema. At each decoding step, only tokens that could lead to a valid parse are allowed. This guarantees syntactically valid structured output. While n-gram blocking prevents repetition, grammar constraints enforce structural rules—the two can be combined for both format compliance and diversity.

A decoding strategy that contrasts the logit outputs from a later, mature transformer layer against an earlier, premature layer within the same model. By amplifying knowledge that emerges only in deeper layers, DoLa surfaces factual information and reduces hallucinations without requiring an external amateur model. This internal contrast mechanism is orthogonal to n-gram blocking and can be applied simultaneously.