Redundancy Penalty

A redundancy penalty operates by modifying the raw prediction scores (logits) of candidate tokens before sampling. If a token or n-gram has already been generated, its logit is reduced by a penalty factor (e.g., repetition_penalty in Hugging Face Transformers). This directly lowers the probability of the model repeating itself, forcing it to explore alternative vocabulary and syntactic structures.

Unlike soft logit penalties, n-gram blocking is a deterministic, hard constraint. It sets the probability of any n-gram (e.g., a 3-gram) that has already appeared in the context to exactly zero.

• Mechanism: Prevents exact repetition of phrases.
• Use Case: Essential for avoiding degenerate loops in text generation.
• Limitation: Can be too rigid, breaking natural syntax if a common phrase is needed legitimately.

In retrieval-augmented generation and extractive summarization, redundancy is penalized during the selection phase, not the generation phase. MMR selects passages by maximizing a linear combination of:

• Relevance: Cosine similarity to the query.
• Novelty: Negative cosine similarity to already-selected passages.

This ensures the final retrieved set is both relevant and diverse, avoiding the inclusion of near-duplicate documents.

Contrastive Decoding penalizes redundancy implicitly by searching for tokens that maximize the probability difference between a strong expert model and a weaker amateur model. The amateur model captures common, repetitive patterns. By subtracting its logits, the expert is steered away from generic, high-frequency tokens and toward more informative, surprising choices that reduce summary redundancy.

Standard beam search often produces sequences that are near-duplicates of each other. Diversity constraints penalize beams that generate tokens already selected by sibling beams within the same group.

• Hamming Diversity: Penalizes token overlap between beams.
• n-gram Diversity: Prevents any n-gram from appearing in more than one beam. This is crucial for generating multiple distinct summaries or translations from a single input.

Commercial API endpoints expose redundancy control through tunable parameters:

• Frequency Penalty: Reduces the probability of a token proportionally to how many times it has already appeared in the text. This penalizes overuse of specific words.
• Presence Penalty: Applies a flat penalty to a token if it has appeared at all, encouraging the model to introduce entirely new concepts rather than just varying word frequency.

A decoding parameter that explicitly penalizes the generation of tokens or phrases that are semantically similar to content already produced. It acts as a hard or soft constraint on the model's output distribution.

• Implementation: Often achieved via logit bias or by modifying sampling probabilities.
• Purpose: Prevents repetitive loops and encourages a more comprehensive coverage of distinct topics.
• Contrast: Unlike a simple frequency penalty, a diversity constraint considers semantic similarity, not just exact token repetition.

A strict decoding strategy that prevents a language model from generating any sequence of 'n' tokens that has already appeared in the preceding context window.

• Granularity: Configurable by setting the value of 'n' (e.g., 2-gram, 3-gram).
• Effect: Forcefully eliminates exact repetitive phrasing at a granular level.
• Limitation: Does not prevent semantic redundancy where different words convey the same meaning. A redundancy penalty is more sophisticated as it operates on semantic embeddings.

A generation technique that improves text quality by searching for tokens that maximize the probability difference between a strong expert model and a weaker amateur model.

• Redundancy Reduction: The amateur model is prone to generic, repetitive patterns. By penalizing tokens the amateur over-predicts, the expert's output becomes more novel and informative.
• Mechanism: Subtracts the amateur's log-probabilities from the expert's.
• Result: Surfaces high-quality, non-repetitive text without external training.

A prompting technique that iteratively refines a summary to increase its information density without increasing length. It directly combats redundancy by forcing the model to pack more unique entities and details into a fixed token budget.

• Process: Starts with an initial sparse summary, then iteratively identifies and fuses missing salient entities while removing uninformative words.
• Goal: Achieve a high entity-per-token ratio.
• Outcome: A summary that is maximally informative and minimally redundant.

The computational process of predicting which words, phrases, or entities in a text are most important. A redundancy penalty relies on accurate salience estimation to avoid penalizing the legitimate repetition of a central, high-importance entity.

• Methods: Often uses graph-based algorithms like TextRank or supervised classifiers.
• Function: Distinguishes between a core theme that must be repeated for coherence and a redundant detail that should be suppressed.
• Integration: The redundancy penalty weight is often modulated by the salience score of the token in question.