Constrained Decoding

Forces the model to generate tokens that conform to a specified context-free grammar (CFG) or Backus-Naur Form (BNF). At each generation step, the set of valid next tokens is restricted to those that can legally extend the partial output according to the grammar rules.

• Guarantees syntactically valid JSON, SQL, or custom DSLs
• Prevents malformed structural errors entirely
• Used in code generation and structured data extraction

Ensures the output includes or excludes specific lexical items—words, phrases, or entity types—at designated positions. The decoder is guided to satisfy these constraints without violating the overall fluency of the text.

• Inclusion constraints: Force specific keywords or named entities to appear
• Exclusion constraints: Blacklist prohibited terminology or sensitive data
• Implemented via finite-state automata that track constraint fulfillment

Applies a token mask at each decoding step based on a target schema, such as a JSON Schema or Pydantic model. The mask zeroes out logits for tokens that would violate the expected data type, field name, or value range.

• Ensures string fields contain only string-valid tokens
• Restricts enum fields to predefined value sets
• Validates numeric ranges and boolean types during generation

Uses regex patterns to dynamically filter the model's output vocabulary at each step. Only tokens that, when appended to the current sequence, keep the full string matchable by the regex are permitted.

• Ideal for enforcing formats like dates, phone numbers, or IDs
• Builds a deterministic finite automaton (DFA) from the regex
• Tracks automaton state to determine valid continuations

Integrates constraint satisfaction directly into the beam search decoding algorithm. Each candidate beam is scored not only on model probability but also on its adherence to structural constraints, pruning beams that violate rules.

• Combines fluency optimization with hard constraint enforcement
• Maintains a fixed number of valid partial hypotheses
• Prevents the model from being forced into low-probability dead ends

Performs real-time validation on each generated token before it is finalized. A lightweight validator checks the token against a set of rules—such as allowed character sets, length limits, or prefix/suffix requirements—and rejects invalid selections.

• Operates as a post-softmax filter before sampling
• Can enforce character-level constraints like ASCII-only output
• Useful for preventing injection attacks in code generation

The core mechanism of constrained decoding that forces generation to adhere to a context-free grammar (CFG) or regular expression. At each token step, the system masks the model's logits, setting the probability of any token that would violate the grammar to negative infinity. This guarantees the final output is a valid member of the target language, such as a parseable JSON object or a syntactically correct SQL query. Tools like Outlines and Guidance compile user-defined schemas into finite-state machines that drive this token-level masking.

A critical distinction in structured generation. JSON Mode (offered by API providers like OpenAI) is a fine-tuned instruction-following behavior that strongly encourages valid JSON but does not mathematically guarantee it. Grammar Masking (true constrained decoding) alters the sampling process itself, providing a 100% syntactic guarantee. For mission-critical pipelines, grammar masking eliminates the need for retry logic and error handling on malformed outputs.

A subtle artifact of tokenization that constrained decoders must address. When a prompt ends mid-token (e.g., prompting for {"name": where the model's tokenizer splits ": into a single token), the model may generate a leading space or invalid character. Token healing automatically rewinds and regenerates the boundary token to ensure the prompt's final token is compatible with the enforced grammar, preventing structural failures at the prompt-completion seam.

The primary open-source tools implementing constrained decoding:

• Outlines: Compiles Pydantic schemas into index-based FSM states for efficient logit masking
• Guidance: Microsoft's templating language that interleaves generation with grammar constraints
• LMQL: A query language that allows constraints to be expressed as logical conditions on generated sequences
• SGLang: A structured generation runtime that optimizes constraint checking for high-throughput serving

Constrained decoding operates at two levels. Lexical constraints enforce surface-level patterns like regex matches or keyword inclusion. Semantic constraints enforce logical validity against a schema—for example, ensuring a generated JSON object has all required fields with correct types. The most powerful systems combine both: a JSON schema defines the semantic structure, while a derived FSM enforces the lexical token sequence that satisfies it.

A lighter-weight alternative to full grammar masking. Instead of zeroing out invalid tokens, logit bias adds a configurable penalty or boost to specific token IDs before sampling. This can discourage (but not prevent) certain outputs. Useful for soft constraints like suppressing specific phrases or encouraging domain terminology, but insufficient for hard structural guarantees where a single invalid token breaks downstream parsers.