Repetition Penalty

Repetition penalty operates directly on the raw logits (unnormalized scores) before the softmax function. For each token already present in the generated sequence, the penalty subtracts a value from its logit or scales it down, making it less likely to be sampled again. This is distinct from frequency penalty, which applies a penalty proportional to how many times a token has appeared. The standard formula applies a multiplicative penalty p where logit_i = logit_i / p if logit_i > 0 and logit_i = logit_i * p if logit_i < 0, preventing sign inversion.

Without a repetition penalty, language models are prone to positive feedback loops where high-probability tokens reinforce themselves. A classic failure mode is infinite repetition of a phrase like 'I am. I am. I am.' The penalty breaks this cycle by progressively reducing the probability of tokens that have already been emitted. This is especially critical in greedy decoding and beam search, where deterministic selection amplifies the model's tendency to get stuck in local probability maxima.

Setting the penalty too high introduces a distinct failure mode: the model avoids not just repeated words but also functionally necessary tokens like 'the', 'a', and 'is'. This produces grammatically broken, nonsensical output. Typical values range from 1.0 (no penalty) to 1.2, with values above 1.5 often degrading quality. The optimal setting is task-dependent:

• Creative writing: 1.1–1.2 for lexical diversity
• Factual summarization: 1.0–1.05 to preserve named entities
• Code generation: 1.0, as repetition of syntax tokens is expected

Advanced implementations extend the penalty beyond single tokens to n-gram sequences. An n-gram repetition penalty (often called no_repeat_ngram_size) prevents any sequence of n consecutive tokens from appearing more than once. This is particularly effective at preventing the model from repeating entire sentences or paragraphs verbatim. For example, setting no_repeat_ngram_size=3 ensures no trigram appears twice, forcing the model to rephrase concepts rather than copy-pasting prior output.

Repetition penalty interacts non-trivially with other decoding parameters:

• Top-k sampling: The penalty is applied before the top-k filter, so penalized tokens may still be excluded if they fall below the threshold
• Top-p (nucleus) sampling: Penalty reshapes the cumulative probability mass, potentially excluding tokens that would otherwise be in the nucleus
• Temperature: High temperature flattens distributions, amplifying the relative effect of the penalty; low temperature sharpens distributions, making the penalty less impactful
• Beam search: Penalty must be applied per-beam to prevent all beams from collapsing to the same repetitive sequence

The scope of repetition tracking is a critical design choice. A full-context penalty tracks all tokens from the beginning of the generation, which can unfairly penalize common words in long outputs. A sliding window penalty only considers the last N tokens, allowing natural lexical recycling over longer distances. Some implementations exclude stop words and punctuation from penalty calculation entirely, recognizing that these function words must repeat frequently for grammatical English. The repetition_penalty_range parameter in many inference engines controls this window size.