Structured Generation

A constrained decoding technique that restricts a language model's token-by-token generation to follow a formal grammar, ensuring syntactically valid output. The grammar, often defined in Extended Backus-Naur Form (EBNF), acts as a real-time filter during the model's sampling process.

• Mechanism: At each generation step, the decoder checks the candidate next tokens against the grammar's production rules, discarding any that would lead to an invalid parse tree.
• Use Case: Guaranteeing outputs are valid JSON, SQL, or arithmetic expressions without relying on the model's latent knowledge of syntax.
• Example: The Guidance and Outlines libraries implement this by integrating a finite-state machine representing the grammar into the model's sampling loop.

A technique for guaranteeing that a large language model's output strictly adheres to a predefined JSON Schema, including data types, required fields, and value constraints. This is often implemented via API parameters (e.g., OpenAI's response_format).

• Mechanism: The model is instructed, either via system prompt or internal API logic, to generate a JSON object that validates against the provided schema. Some systems perform schema-aware decoding to bias generation toward valid keys and value types.
• Key Elements: Enforces type (string, number, boolean, object, array), required properties, enum value lists, and nested properties.
• Result: Creates a reliable data contract between the LLM and downstream application code, eliminating parsing errors.

A prompt engineering pattern where a pre-formatted text skeleton with placeholders is provided to guide the model. This is a few-shot learning technique that explicitly teaches the desired structure.

• Structure: The prompt includes a clear example, often with XML-like tags or a JSON block, showing the exact format. Example: <name>Placeholder Name</name><role>Placeholder Role</role>
• Process: The model infills the placeholders based on the task context. This is less rigid than grammar-based decoding but highly effective with capable models.
• Best Practice: Combining the template with a clear instruction ("Fill in the following XML template") and type hints (e.g., <!-- integer -->) significantly improves accuracy.

A broad family of inference-time algorithms that bias or restrict token generation to enforce output patterns. Grammar-based decoding is a subset. Other methods include:

• Token Masking: Prohibiting certain tokens from being generated at specific positions.
• Regular Expression Matching: Constraining the output string to match a regex pattern.
• Finite-State Machine Decoding: Guiding generation through a predefined state graph representing valid output sequences.
• Purpose: These algorithms provide a data format guarantee at the sampling level, making structured generation robust even for models not explicitly fine-tuned for the task.

The application of automated scripts to parse, validate, and normalize a raw model response after generation. This is a safety net when other enforcement mechanisms are not available or fail.

• Structured Output Parsing: Using a library like Pydantic or a JSON parser to extract data, catching syntax errors.
• Output Validation: Checking the parsed data against a response schema for semantic correctness (e.g., age must be > 0).
• Output Normalization: Transforming the data into a canonical format (e.g., converting all dates to ISO 8601).
• Self-Correction Loop: On validation failure, the invalid output can be fed back to the model with an error message for a retry, enabling recursive error correction.

An approach where a formal schema (e.g., JSON Schema, Protobuf) is provided as a key part of the model's context to explicitly guide the structure and content of its output. This is a form of in-context learning.

• Process: The schema is injected into the system prompt or few-shot examples. The model is instructed to "generate an output matching this schema."
• Advantage: More flexible than hard decoding constraints, allowing the model to understand semantic relationships and required fields described in the schema.
• Combination: Often used in conjunction with function calling instructions, where the schema defines the parameters for a tool the model must call.

A constrained decoding technique that restricts a model's token-by-token generation to follow a formal grammar defined in a notation like EBNF (Extended Backus-Naur Form). The decoder acts as a parser, only allowing tokens that result in a syntactically valid sequence according to the grammar. This is a powerful method for guaranteeing outputs in formats like JSON, SQL, or custom DSLs.

• Key Benefit: Provides absolute syntactic guarantees.
• Implementation: Libraries like Guidance or Outlines apply this during inference.

The downstream process of programmatically extracting and validating data from a model's response based on a specified format. While structured generation aims to produce parseable output, parsing is the act of consuming it. This involves:

• Deserialization: Converting a JSON/XML string into native data structures (e.g., Python dict).
• Validation: Checking the deserialized data against a schema.
• Error Handling: Managing cases where the output is malformed despite generation constraints.

A formal specification that defines the exact structure, data types, and constraints for a model's output. It serves as the data contract between the LLM and the consuming application. Common implementations include:

• JSON Schema: The most prevalent standard for LLM APIs.
• Pydantic Models: Used in Python ecosystems to define and validate schemas.
• Protocol Buffers / gRPC: For high-performance, typed service contracts. The schema is often injected into the prompt or passed separately to the model API.

A broad family of inference-time algorithms that bias or restrict a model's token generation to enforce specific output patterns. This is the underlying mechanism for many structured generation techniques. Methods include:

• Grammar-Based Decoding (see above).
• Regex/Keyword Constraints: Forcing the output to match a pattern.
• Schema-Aware Decoding: Dynamically adjusting logits based on a live schema state. These techniques operate on the model's logits before sampling, ensuring the final output adheres to hard or soft constraints.

A pre-formatted text skeleton provided within the prompt, containing placeholders (e.g., {{date}}, {{summary}}) that guide the model to fill in specific information in a consistent structure. This is a fundamental prompt engineering pattern for structured generation.

Example Prompt:

codeCopy
Extract the key details. Use this format:
Company: {{company_name}}
Date: {{date}}
Amount: {{amount}}

The model is expected to generate text that fits precisely into this template, making subsequent deterministic parsing (e.g., via regex) straightforward.