Constrained Decoding

The core technique applies a binary mask over the model's raw logits before the softmax function. Tokens that would lead to an invalid state according to the target grammar are assigned a value of -∞, effectively setting their selection probability to zero. This guarantees that the autoregressive generation never deviates from the specified formal language.

Constraints are defined using formal specifications such as JSON Schema, Regular Expressions, or Context-Free Grammars (CFG). These schemas are compiled into a deterministic finite automaton (DFA) that tracks the valid state at each generation step. Common formats include:

• GBNF (GGML BNF): Used in llama.cpp for low-level grammar enforcement
• Pydantic Models: Python class definitions compiled into JSON schema constraints
• Lark Grammars: For complex nested structures like SQL or custom DSLs

The system maintains a state machine that advances with each generated token. After every token selection, the current state is updated to reflect the partial output's position within the grammar. This allows the mask to dynamically adapt, permitting only tokens that represent valid continuations from the current parse state. This is critical for enforcing nested structures like matching brackets or closing XML tags.

Constrained decoding operates as a post-logit transformation that is compatible with all standard sampling methods. The mask is applied before temperature scaling, top-k filtering, or nucleus (top-p) sampling. This ensures that creative sampling techniques do not compromise output validity. The constraint layer acts as a non-negotiable gate that precedes any stochastic selection.

Modern implementations like Outlines and Guidance compile grammars into efficient index-based state machines that add minimal latency overhead. The mask is applied as a vectorized operation on the GPU, typically adding less than 5% overhead to per-token generation time. Pre-compiled grammars can be cached and reused across multiple generation requests, amortizing the initial parsing cost.

While JSON mode is the most common application, constrained decoding enables reliable generation of:

• SQL queries with exact table and column references
• Python code that is guaranteed to be syntactically valid
• Chemical formulas in SMILES notation
• Mathematical expressions in LaTeX
• Custom DSL outputs for workflow engines This eliminates the need for post-hoc parsing and retry logic in production pipelines.

A programmatic policy or mechanism that constrains an AI agent's behavior to prevent harmful, off-policy, or unsafe actions and outputs. Guardrails operate at multiple levels of the agent stack.

• Input Guardrails: Filter or sanitize prompts before they reach the model
• Output Guardrails: Validate and filter generated content before delivery
• Tool Guardrails: Restrict which functions an agent can call and with what parameters

Constrained decoding serves as a structural guardrail by preventing syntactically invalid outputs from being generated in the first place.

A control point in an agentic workflow that requires explicit validation or approval before a high-stakes tool call or state-changing operation is executed. Action gates are the last line of defense after output validation.

• Human-in-the-Loop Gates: Route critical decisions to a human operator for manual approval
• Automated Policy Gates: Use rules engines like OPA to verify compliance automatically
• Confidence-Based Gates: Block actions when model confidence falls below a threshold

While constrained decoding ensures syntactic correctness, action gates ensure semantic safety of the intended operation.

The process of removing or neutralizing potentially dangerous content from an agent's generated output before delivery or execution. Sanitization addresses threats that constrained decoding cannot prevent.

• Code Sanitization: Strips executable code, SQL injection attempts, or shell commands
• PII Redaction: Detects and masks personally identifiable information like emails and SSNs
• HTML/JS Stripping: Removes cross-site scripting vectors from generated content

Constrained decoding prevents format errors; sanitization prevents content-based attacks that are syntactically valid but semantically dangerous.

A secondary language model or classifier that evaluates the primary agent's outputs for correctness, safety, or alignment, providing a feedback signal for rejection or refinement. Critic models add a semantic review layer on top of structural constraints.

• Factual Consistency Critics: Use NLI models to detect hallucinations
• Safety Critics: Classify outputs for toxicity, bias, or policy violations
• Task-Specific Critics: Verify domain-specific requirements like legal citation accuracy

Constrained decoding guarantees the output is well-formed; a critic model verifies it is well-reasoned.

A quantitative metric that estimates the degree of factual inconsistency in a generated response, often derived from semantic similarity or natural language inference models. This score feeds into output validation pipelines.

• Semantic Entropy: Measures uncertainty across meaning clusters rather than token sequences
• Chain-of-Verification (CoVe): Self-checks output by generating and answering fact-checking questions
• Citation Grounding: Verifies each claim against a retrieved source document

Constrained decoding ensures structural fidelity; hallucination scoring ensures factual fidelity. Both are essential for production agent reliability.