Diversity Constraint

A scoring function that explicitly reduces the probability of selecting a token or passage if it is highly similar to content already present in the generated sequence. This operates by computing cosine similarity between candidate embeddings and previously selected items, then subtracting a weighted penalty from the logits before softmax normalization.

• Prevents repetitive phrasing in long-form generation
• Applied at both token-level and passage-level in RAG pipelines
• Typical penalty coefficients range from 0.1 to 0.5

An iterative selection algorithm that balances relevance to the query against dissimilarity to already-selected items. MMR constructs a diverse result set by greedily choosing the next passage that maximizes a linear combination of query relevance and novelty.

• Formula: MMR = λ * Relevance(doc, query) - (1-λ) * max Similarity(doc, selected)
• λ parameter controls the diversity-relevance tradeoff
• Originally developed for text summarization at Carnegie Mellon

A hard constraint during decoding that prevents the model from generating any sequence of n tokens that has already appeared in the context window. Unlike soft penalties, n-gram blocking is deterministic and absolute.

• Common settings: 2-gram (bigram) or 3-gram (trigram) blocking
• Eliminates exact phrase repetition without affecting semantic variety
• Implemented natively in Hugging Face Transformers via no_repeat_ngram_size

A generation technique that amplifies desirable behaviors by searching for tokens that maximize the probability difference between a strong expert model and a weaker amateur model. The amateur model captures common but uninteresting patterns, and the contrastive objective suppresses them.

• Expert model: larger, more capable LLM
• Amateur model: smaller, less capable variant
• Naturally suppresses generic, high-frequency completions
• Produces more informative and varied outputs without explicit diversity tuning

Stochastic decoding parameters that control the randomness of token selection, directly influencing output diversity. Higher temperature flattens the probability distribution, giving less likely tokens a greater chance. Top-p (nucleus) sampling truncates to the smallest set of tokens whose cumulative probability exceeds p.

• Temperature > 1.0: increases diversity, risks incoherence
• Temperature < 1.0: sharpens distribution, more deterministic
• Top-p = 0.9: considers only tokens in the 90% probability mass
• Combined use prevents both repetition and degeneration

Decoding by Contrasting Layers (DoLa) contrasts the logit outputs from a mature later transformer layer against an earlier premature layer. The premature layer captures surface-level linguistic patterns, while the mature layer encodes factual knowledge. Subtracting the premature logits amplifies the factual signal.

• No external amateur model required
• Operates entirely within a single model's internal representations
• Proven to reduce hallucinations while maintaining diverse, factual output
• Effective for open-ended generation tasks requiring both accuracy and variety

A foundational algorithm for information retrieval and summarization that selects passages by balancing two competing factors: relevance to the user's query and novelty against already-selected content. It uses a tunable lambda parameter to trade off accuracy for diversity, directly penalizing redundancy. MMR is a classic, pre-neural implementation of the diversity constraint principle, widely used in extractive summarization pipelines to ensure a summary covers different facets of a topic rather than repeating the same point.

A scoring mechanism that explicitly reduces the weight or probability of a token or passage being selected if it is highly similar to content already present in the generated summary or retrieved set. In neural text generation, this is often implemented via n-gram blocking or by modifying logits during decoding. By suppressing repetitive phrases, the redundancy penalty directly enforces a diversity constraint, forcing the model to explore semantically varied alternatives and produce a more information-dense, non-repetitive output.

A generation technique that improves text quality by searching for tokens that maximize the probability difference between a strong expert model and a weaker amateur model. The core insight is that desirable behaviors like factual accuracy and lexical diversity are amplified in the expert, while repetitive, dull patterns are common to both. By penalizing tokens the amateur finds likely, contrastive decoding acts as a powerful diversity constraint, yielding more interesting and varied language than standard sampling methods.

A prompting technique that iteratively refines a summary to increase its information density without increasing its length. Starting with a sparse initial summary, the model is prompted to identify and fuse 1-3 missing salient entities into each new iteration, explicitly packing more unique concepts into a fixed token budget. CoD is a direct application of a diversity constraint at the prompt level, trading stylistic fluency for a higher concentration of distinct, non-redundant facts.

A decoding strategy that prevents a language model from generating any sequence of 'n' tokens that has already appeared in the context. By setting a hard constraint against exact repetition, n-gram blocking is a simple but effective diversity constraint that eliminates repetitive phrasing at a granular level. It is commonly used in summarization and creative text generation to force the model to use different lexical choices and syntactic structures, increasing the overall variety of the output.

A metric and content strategy for providing unique, substantive value beyond what an AI model already knows from its training data. In the context of summarization, content optimized for high information gain inherently contains diverse, novel facts that are less likely to be redundant with the model's pre-existing knowledge. This external diversity constraint ensures that when a model summarizes such content, the output is rich with new entities and insights rather than generic, commonly known statements.