Logit Bias

Logit bias operates by modifying the unnormalized prediction scores (logits) before the softmax function converts them into probabilities.

• A positive bias (+10.0) adds to the logit, exponentially increasing the token's probability.
• A negative bias (-10.0) subtracts from the logit, effectively banning the token.
• The modification is absolute and deterministic, not a suggestion.

This is distinct from prompt engineering, which relies on the model's semantic interpretation.

The most common use case is applying a large negative bias (e.g., -100) to prevent specific tokens from ever being sampled.

• Safety Filtering: Ban tokens associated with profanity, hate speech, or personally identifiable information (PII).
• Format Enforcement: Suppress narrative tokens to force structured outputs like JSON or XML.
• Stop Sequence Simulation: Ban all tokens except a specific stop token to terminate generation precisely.

This provides a hard guarantee that prompt-based safety instructions cannot.

Positive logit bias can gently nudge the model toward a specific lexicon or terminology set without rigid constraints.

• Brand Language: Increase the probability of trademarked product names or specific corporate jargon.
• Domain Adaptation: Promote technical terms from a target field like medicine or law to shift the output's register.
• Sentiment Tuning: Apply a slight positive bias to tokens associated with positive sentiment lexicons.

This technique is often paired with controlled generation and grammar-constrained decoding for robust pipelines.

The effectiveness of logit bias is entirely dependent on the model's tokenizer. A word must be represented as a distinct token in the vocabulary to be biased.

• Subword Pitfalls: A single word like "unhappiness" might be tokenized as ["un", "happiness"]. Biasing the token "unhappiness" will have no effect if it doesn't exist in the vocabulary.
• Leading Spaces: Many tokenizers encode spaces as part of the token (e.g., " word" vs "word"). You must bias the exact token ID.
• Tooling Required: Always use the model's tokenizer to encode the target string and retrieve the precise integer token ID before applying the bias.

These two control mechanisms operate on fundamentally different layers of the model stack.

• Prompting: Influences the model's internal world state and attention patterns. It is probabilistic and interpretable by the model.
• Logit Bias: A post-hoc mathematical override applied to the output projection layer. It is deterministic and invisible to the model's reasoning process.

For critical compliance, logit bias provides a hard constraint that prompting cannot guarantee. For nuanced context, prompting is superior.

Major LLM providers expose logit bias as a parameter in their completion endpoints.

• OpenAI / Azure: Accepts a logit_bias map of token ID to bias value (range: -100 to 100).
• Google AI / Vertex AI: Uses a LogitBias map in the generation config.
• Open-Source (Hugging Face): Implemented manually by modifying the logits tensor before the softmax call in the generation loop.

A common pattern is to combine a high temperature for creativity with a negative logit bias to strictly exclude undesirable content.