Guided Decoding

The fundamental mechanism that physically prevents invalid token generation. Token masking dynamically sets the probability of out-of-schema tokens to negative infinity (or zero) before the softmax step, making their selection impossible. Logit bias modifies the raw scores (logits) of specific tokens to suppress or encourage them, useful for nudging the model toward structural tokens like { or [ without hard constraints. This operates at the lowest level of the decoding loop.

Uses a formal grammar, such as a Context-Free Grammar (CFG) or GBNF Grammar, to define the universe of valid outputs. The grammar is compiled into a Finite State Machine (FSM) that tracks the current valid state. At each decoding step, the FSM provides the exact set of permissible next tokens, guaranteeing the final output is syntactically parseable. This is the most robust method for generating complex nested structures like JSON.

Tightly couples generation with validation libraries like Pydantic or JSON Schema. The target schema is translated into a set of constraints that guide the decoder. This ensures not just valid JSON, but type-safe, semantically correct output. For example, a Pydantic model defining age: int and name: str will force the model to generate a number for the age field and a string for name, preventing type errors before the response is complete.

Setting the temperature parameter to zero eliminates randomness from the sampling process. The model always selects the highest-probability token. When combined with guided decoding, this produces perfectly reproducible, deterministic structured outputs. This is critical for production API pipelines where non-determinism can cause schema drift and downstream processing failures.

A stop sequence is a predefined string (e.g., </json>, ###) that signals the model to halt generation immediately. In structured output, this prevents the model from generating trailing text or commentary after the valid data structure closes. It acts as a structural boundary, ensuring the output is cleanly terminated and ready for parsing without post-hoc trimming.

Combines reasoning with formatting by requiring the model to output its step-by-step logic in a parseable structure. The ReAct Agent Format is a prime example, enforcing a strict interleaving of Thought:, Action:, and Observation: tokens. This constrains the reasoning trace itself, making it auditable and machine-readable, and prevents the model from drifting into unstructured rambling before delivering the final structured answer.

The definitive use case for guided decoding. When an LLM must invoke an external API, its output must be a valid JSON object with the correct function name and arguments.

• Function Calling: Models output a structured payload like {"name": "get_weather", "arguments": {"city": "London"}}.
• Schema Adherence: A JSON Schema is passed to the generation process, and token masking ensures curly braces, keys, and value types are syntactically perfect.
• Error Elimination: Prevents the model from adding trailing commas, unescaped quotes, or explanatory text before the JSON, which would break a JSON.parse() call.

Transforming unstructured text (emails, legal documents, medical records) into a predefined Pydantic model or database row. Guided decoding guarantees the extracted fields exist and have the correct data types.

• Entity & Relation Extraction: Force the output to be a list of {"entity": "...", "type": "..."} objects.
• Slot Filling: Populate a template like {"departure": "...", "arrival": "...", "date": "..."} from a user's natural language query.
• Type Safety: If the schema defines a field as an integer, the model is physically prevented from generating a string, ensuring downstream ETL pipelines don't break.

Autonomous agents rely on structured internal monologues. The ReAct Agent Format requires strict alternation between Thought:, Action:, and Observation: tokens.

• State Machine Enforcement: A Finite State Machine (FSM) tracks the agent's progress. After Action:, the FSM only allows the generation of a valid tool name.
• Preventing Hallucination: The model cannot invent a new reasoning step format or skip directly to a final answer without executing the required tool.
• Deterministic Parsing: Guarantees the orchestration framework can reliably parse the agent's intended next step without regex hacks.

Generating executable code requires strict adherence to a language's Context-Free Grammar (CFG). Guided decoding ensures every generated line is syntactically valid.

• AST Validation: The generation process can be constrained by an Abstract Syntax Tree (AST), ensuring parentheses are balanced and keywords are correctly placed.
• Domain-Specific Languages (DSLs): Force the LLM to output only valid commands for an internal DSL, preventing runtime errors in automated workflows.
• GBNF Grammar: Tools like llama.cpp use GBNF Grammar definitions to constrain open-source models to produce valid programming language syntax.

When the output must be a single, machine-readable label from a finite set, guided decoding eliminates post-processing guesswork.

• Logit Bias: The probabilities of all tokens except the valid class labels (e.g., "positive", "negative", "neutral") are set to negative infinity.
• No Explanations: The model is physically blocked from outputting "The sentiment is..." or any other natural language preamble.
• Downstream Reliability: A classification pipeline receives exactly one of the allowed strings, enabling deterministic routing and alerting.

Combining reasoning with reliability. The model is forced to output its step-by-step logic in a parseable format before delivering the final answer.

• Chain-of-Thought Structuring: The output schema might be {"reasoning_steps": ["..."], "final_answer": "..."}.
• Dual Validation: The reasoning is human-readable for debugging, while the final answer is machine-parseable for the application.
• Hallucination Mitigation: Constraining the final answer to a specific schema (e.g., a number or a boolean) prevents the model from generating a verbose, incorrect explanation as the final output.

The direct enforcement of a formal grammar during the token-by-token generation process. Unlike post-processing, this method uses a Context-Free Grammar (CFG) or Finite State Machine (FSM) to define the set of all valid next tokens at each step.

• Mechanism: Dynamically masks invalid token probabilities before sampling.
• Key Benefit: Provides a mathematical guarantee of syntactic validity.
• Contrast: Differs from simple JSON mode prompting by enforcing rules at the logit level.

A declarative vocabulary for annotating and validating JSON documents. In structured output, a JSON Schema serves as the blueprint that defines the expected object structure, required fields, and data types.

• Role: Acts as the target specification for constrained generation.
• Features: Supports enum restrictions, pattern validation with regex, and nested object definitions.
• Integration: Libraries like instructor and outlines translate these schemas directly into generation constraints.

A low-level decoding technique that dynamically sets the probability of invalid tokens to negative infinity (or zero after softmax) at each generation step. This physically prevents the model from selecting tokens that would break the structural contract.

• Implementation: Uses a mask matrix applied to the logits before sampling.
• Efficiency: Adds minimal latency overhead compared to beam search methods.
• Use Case: The foundational mechanism behind libraries like llama.cpp's GBNF grammar support.

An abstract computational model used to track the current valid state of an output sequence. In guided decoding, an FSM is compiled from a schema or grammar to determine the exact set of permissible next tokens.

• States: Represent progress through the structure (e.g., 'expecting key', 'inside string').
• Transitions: Triggered by specific token generations.
• Advantage: Extremely fast index-based lookups for determining valid continuations.

A set of recursive production rules that formally define all possible valid strings in a language. GBNF Grammar is a popular notation for defining these rules in local inference engines.

• Structure: Consists of terminals (literal tokens) and non-terminals (rules).
• Power: Can express complex nested structures like code or mathematical expressions.
• Tooling: Used directly by llama.cpp and Outlines to constrain open-source models.

A decoding strategy that explores multiple generation paths simultaneously while pruning branches that violate predefined lexical constraints. It finds the most probable valid output rather than just the first valid one.

• Mechanism: Maintains a beam of top-k candidates and discards invalid ones.
• Trade-off: Higher computational cost than greedy masking but often yields higher quality.
• Application: Useful when strict format adherence must be balanced with semantic quality.