Grammar-Constrained Decoding

The core mechanism that forces a language model to output text conforming to a predefined formal grammar, such as a JSON schema, XML DTD, or custom BNF specification. During autoregressive generation, the decoder masks the logits of all tokens that would violate the grammar rules, ensuring every generated token maintains syntactic validity. This transforms probabilistic text generation into a deterministic structural guarantee, eliminating malformed outputs entirely.

The technical implementation that applies grammar constraints in real-time during token generation. At each decoding step, the system:

• Evaluates the current partial output against the grammar's finite-state automaton
• Identifies the set of valid next tokens permitted by the grammar
• Applies a mask to the model's raw logits, setting invalid token probabilities to negative infinity
• Samples only from the remaining valid token set This occurs at inference time with no model retraining required.

The most common production use case where grammar constraints guarantee valid JSON output matching a specified schema. The grammar defines required fields, data types, nesting structures, and array constraints. Models using grammar-constrained decoding produce parseable JSON 100% of the time, eliminating the need for retry logic, regex extraction, or error handling for malformed outputs. Critical for API integrations, function calling, and structured data extraction pipelines.

The formal language theory foundation using context-free grammars (CFGs) expressed in Extended Backus-Naur Form (EBNF). The grammar defines production rules that specify valid token sequences. Implementations like llama.cpp's GBNF or Outlines compile these grammars into efficient finite-state machines that guide decoding. This approach supports arbitrary nested structures, recursive definitions, and complex syntactic patterns beyond simple regex constraints.

Ecosystem of open-source tools implementing grammar-constrained decoding:

• Outlines: Python library using finite-state machines for structured generation
• Guidance: Microsoft's templating language with grammar enforcement
• LMQL: Query language combining prompting with constraints
• llama.cpp: GBNF grammar support for local model inference
• SGLang: Efficient structured generation with RadixAttention Each library compiles grammars into efficient automata for real-time token masking.

Grammar constraints eliminate wasted tokens from malformed outputs and retries. Without constraints, models may generate invalid JSON requiring multiple regeneration attempts, consuming 2-5x the necessary tokens. Grammar-constrained decoding guarantees first-pass validity, reducing API costs and latency. Additionally, constraints prevent the model from generating explanatory text or conversational filler within structured outputs, ensuring every token contributes directly to the data payload.

A broad category of techniques that steer the output of a language model by manipulating its internal logits or applying hard constraints. Unlike prompt engineering, which relies on instruction-following, controlled generation guarantees adherence to structural or stylistic rules.

• Logit manipulation: Adjusting token probabilities before sampling
• Hard constraints: Forcing valid tokens via masking
• Soft constraints: Biasing probabilities toward preferred tokens
• Use cases: JSON output, sentiment control, topic steering

A parameter that modifies the raw prediction scores (logits) of specific tokens before sampling. Developers can forcefully increase or decrease the probability of certain words appearing in generated output.

• Positive bias: Boosts token likelihood (e.g., +10 to "Paris")
• Negative bias: Suppresses token likelihood (e.g., -100 to ban a word)
• Granularity: Applied at the token-ID level, not the word level
• Limitation: Cannot enforce complex structural rules like JSON validity

A decoding strategy that prevents a language model from generating any sequence of 'n' tokens that has already appeared in the context. This eliminates repetitive phrasing at a granular level.

• 1-gram blocking: No repeated individual tokens
• 3-gram blocking: No repeated three-word phrases
• Common use: Preventing degenerate repetitive loops in open-ended generation
• Trade-off: Overly aggressive blocking can produce unnatural, stilted text

A generation technique that improves text quality by searching for tokens that maximize the probability difference between a strong expert model and a weaker amateur model. The amateur model captures undesirable patterns, and the expert-amateur gap surfaces high-quality tokens.

• Expert model: Full-size, capable language model
• Amateur model: Smaller, less capable version of the same model
• Effect: Amplifies desirable behaviors like factuality and coherence
• Key paper: Li et al., 2022

A decoding strategy that contrasts the logit outputs from a later, mature transformer layer against an earlier, premature layer to surface factual knowledge. Unlike contrastive decoding, DoLa uses a single model and exploits the fact that factual knowledge emerges in later layers.

• No external model required: Uses internal layer differences
• Premature layer: Captures surface-level language patterns
• Mature layer: Contains grounded factual knowledge
• Benefit: Reduces hallucinations without additional compute overhead

A post-processing approach that validates generated text against a schema after generation, retrying with error feedback if validation fails. Unlike grammar-constrained decoding, this is a reactive rather than proactive strategy.

• Schema validation: JSON Schema, Pydantic models, regex patterns
• Retry loop: On failure, feed error message back to model
• Cost implication: Failed generations still consume tokens
• Advantage: Works with any model, no special decoding infrastructure needed