Structured Generation

JSON Schema is a declarative language for annotating and validating JSON documents. When used for structured generation, it provides a strict contract that the model's output must fulfill. The schema defines required properties, data types (string, integer, array, object), and constraints like enum values, string patterns (regex), and numerical ranges.

• How it works: The schema is provided to the model as part of the system prompt or via a dedicated API parameter (e.g., response_format). The model is instructed to generate output that is both syntactically valid JSON and semantically valid against the provided schema.
• Example: A schema for a user profile might require a name (string), age (integer, minimum 0), and status (string, enum: ['active', 'inactive']). The model cannot output a boolean for age or a status not in the list.
• Primary Use: Guaranteeing API request/response shapes and ensuring data interoperability between systems.

Pydantic is a Python library that uses Python type annotations to perform runtime data validation and serialization. For structured generation, a Pydantic model class defines the expected output structure. The raw text from the language model is parsed and then instantiated into the Pydantic model, which automatically validates all fields.

• How it works: Developers define a class inheriting from pydantic.BaseModel with typed fields. After generation, a parser (like OpenAI's PydanticOutputFunctionsParser) extracts the text, attempts to create an instance of the model, and raises a ValidationError if the data is invalid.
• Key Advantage: Deep integration with Python development tools (IDE autocompletion, mypy) and support for complex custom validators and field relationships.
• Example: A ToolCall model with fields name: str, arguments: Dict[str, Any], and a custom validator to ensure arguments matches the tool's own JSON Schema.

Grammar constraints restrict the language model's token-by-token generation to only produce sequences that are valid according to a formal grammar, typically a Context-Free Grammar (CFG). This provides the strongest guarantee of syntactic correctness.

• How it works: A grammar (e.g., in GBNF format for Llama.cpp) is compiled into a state machine. During autoregressive decoding, at each step, the model's logits are masked so only tokens that lead to a grammatically valid continuation can be selected.
• Difference from Schema Guidance: While JSON Schema guides the semantics, grammar constraints control the syntax at the character/token level. It can guarantee valid JSON, XML, or even code in a specific programming language.
• Primary Use: Ensuring output is parseable by strict parsers without any post-hoc "fixing," which is critical for generating code, configuration files, or API calls.

The function calling API pattern is a specialized form of structured generation where the model is presented with a list of callable tools/functions, each defined by a name, description, and parameters schema. The model's task is to output a structured choice to call one (or more) of these functions with the correct arguments.

• How it works: Instead of generating free text, the model outputs a JSON object with fields like tool, tool_input, or a standardized array of tool_calls. The parameters within tool_input must conform to the JSON Schema for that specific tool.
• Key Benefit: It seamlessly bridges natural language intent ("book a flight to NYC") to a structured API call (book_flight(destination="JFK")).
• Example: OpenAI's Chat Completions API has a dedicated tools parameter. The model's response includes a tool_calls array containing the function name and arguments object.

Output parsers are post-processing components that transform raw model text into a structured format. Templating pre-defines the response structure with placeholders for the model to fill. These are often used together for lighter-weight structured generation.

• Parsers: Libraries like LangChain's OutputParser take the text and apply rules (regex, XML tag extraction, JSON parsing) to create a structured object (e.g., a list, a dictionary). They handle cases where the model is instructed to format its answer in a specific way.
• Templating: A prompt template explicitly shows the model the required format using examples, XML tags, or other delimiters (e.g., "<name>John Doe</name><age>30</age>"). This is less rigid than schema enforcement but highly effective for simple structures.
• Use Case: Extracting a list of entities from a paragraph or formatting a summary into specific bullet points without the overhead of a full JSON Schema.

A validation layer is a dedicated software component that sits between the language model and the consuming application. Its sole responsibility is to enforce type safety and business logic rules on the generated output.

• How it works: This layer receives the model's raw or initially parsed output. It validates it against a data contract (e.g., a JSON Schema, Protobuf definition, or Pydantic model). Invalid data triggers a retry, a fallback, or an error.
• Key Concepts:
  • Type Enforcement: Guaranteeing an id field is an integer, not a string.
  • Field Constraints: Ensuring a percentage is between 0 and 100.
  • Contract Enforcement: Guaranteeing the entire data exchange follows the API specification.
• Primary Use: Creating robust, production-grade systems where incorrect data types from the model would cause downstream application failures. It is the final, essential guardrail.

JSON Schema is a declarative language for validating the structure and data types of JSON documents. It provides a machine-readable contract that defines required properties, allowed data types (string, number, boolean, array, object), and constraints like minimum/maximum values or regular expression patterns for strings. In structured generation, the schema is provided to the language model as part of the system prompt or via a dedicated parameter (like response_format), instructing it to generate output that validates against this schema. This is the foundational standard for defining the expected shape of AI-generated API calls and parameters.

Pydantic models are Python classes that use type annotations to define and enforce data schemas at runtime. They provide a programmatic, object-oriented way to achieve structured generation by:

• Defining fields with precise types (e.g., str, int, List[float]).
• Adding custom validators and field constraints (e.g., Field(gt=0), regex patterns).
• Automatically parsing raw JSON or dict output from an LLM into a validated Python object.
• Serializing the validated object back to JSON for API calls. Frameworks like LangChain and Instructor use Pydantic models to wrap LLM calls, guaranteeing the output matches the defined class structure and enabling full IDE support with type hints and autocompletion.

Type enforcement is the runtime or compile-time verification that data values conform to declared type definitions (e.g., string, integer, boolean, custom object). In the context of structured generation, it ensures the fields in an AI model's output are not just present but are of the correct type. This prevents critical errors where an API expects an integer but receives a numeric string, or where a boolean true is sent as the string "true". Enforcement is typically performed by a validation layer that uses a schema (JSON Schema, Pydantic) to check and often coerce types before the data is used in downstream logic or external calls.

Grammar constraints are formal rules, often defined as a Context-Free Grammar (CFG) or a regular expression, that restrict a language model's output to a syntactically valid set of strings. While JSON Schema defines the structure of data, grammar constraints operate at the token level during the model's decoding process. Tools like Guidance or LMQL use grammars to force the model to generate only text that matches the allowed pattern (e.g., a valid JSON string, a SQL query, a function call with correct parentheses). This provides a stronger, more deterministic guarantee than schema guidance alone, as it physically prevents the model from generating invalid syntax.

Output parsing is the process of converting the raw, unstructured text output from a language model into a structured, machine-readable format. This is a critical post-processing step when a model is not explicitly using a structured generation mode. A parser must:

1. Extract relevant data from potentially verbose or free-form text.
2. Map extracted values to a predefined schema.
3. Handle and clean common formatting issues (extra whitespace, markdown code fences). Parsers range from simple regular expressions to sophisticated libraries that use a second, smaller LLM to reformat the output. The goal is to achieve reliability despite the model's inherent unpredictability, bridging the gap to type-safe systems.

JSON mode is a dedicated configuration parameter (e.g., response_format={ "type": "json_object" } in the OpenAI API) that instructs a language model to guarantee its response will be valid, parseable JSON. When enabled, the model is internally constrained to generate tokens that form a complete JSON object. It is a simpler, more robust alternative to asking the model in a system prompt to "output JSON." Key implications:

• The model's output is guaranteed to be parsable by json.loads().
• The user must also provide a schema (via the prompt) to define the contents of that JSON object.
• It significantly reduces the chance of formatting hallucinations like truncated braces or invalid escape sequences.