GBNF Grammar

GBNF uses a Context-Free Grammar (CFG) notation to define all valid output strings. Rules are expressed as recursive production rules, providing a strict syntactic contract for the model.

• Root Rule: Every grammar starts with a root rule defining the top-level structure.
• Terminals: Literal strings and character ranges (e.g., [a-z]) define valid tokens.
• Non-Terminals: Named rules (e.g., json-object) allow recursive composition.
• Alternation: The pipe | operator defines logical OR branches for valid sequences.

During decoding, the GBNF grammar is compiled into a Finite State Machine (FSM). At each step, the FSM calculates the set of permissible next tokens and applies a token mask.

• Logit Bias: The probabilities of all invalid tokens are set to negative infinity.
• Guaranteed Validity: The model is physically prevented from sampling an out-of-schema token.
• Zero Overhead: Masking happens natively in the C++ inference engine for minimal latency impact.

GBNF is the primary mechanism for forcing JSON output in local inference engines. A grammar can precisely define the structure, required keys, and data types of a JSON object.

• Type Safety: Enforce string, number, boolean, and null types.
• Required Fields: Define mandatory keys within an object.
• Array Constraints: Specify the element type and cardinality of arrays.
• Example: A grammar can enforce the exact { "name": string, "age": integer } schema.

Beyond JSON, GBNF enables the creation of custom Domain-Specific Languages (DSLs) for specialized agent outputs. This allows for more token-efficient and parseable formats.

• ReAct Format: Enforce the Thought: ... Action: ... structure for agentic loops.
• Code Generation: Constrain output to a specific programming language syntax.
• Tool Calling: Define a precise grammar for function names and arguments.
• Efficiency: Custom DSLs often require fewer tokens than equivalent JSON.

GBNF is a core feature of the llama.cpp inference engine, making structured output accessible on consumer hardware. The grammar is passed as a string parameter to the API.

• Server API: Use the grammar field in the /completion endpoint.
• Offline Usage: Pass the grammar string directly to the llama_eval() function.
• Grammar Caching: Compiled FSMs are cached to avoid re-parsing on subsequent calls.
• Compatibility: Works with all GGUF-quantized models supported by the runtime.

While JSON Schema defines the what (the data contract), GBNF defines the how (the token-level path to get there). GBNF is a lower-level, more deterministic mechanism.

• JSON Schema: Validates output after generation. Relies on model instruction-following.
• GBNF: Physically constrains generation during decoding. Guarantees syntactic validity.
• Combined Approach: Use a library to compile a JSON Schema into a GBNF grammar for the strongest guarantee.
• Trade-off: GBNF is rigid; it cannot gracefully recover from a semantic error mid-stream.

The computational engine that executes a GBNF grammar. During generation, a Finite State Machine is compiled from the grammar rules. Each state represents a valid point in the output sequence. The FSM tracks the current state and provides the set of permissible next tokens. As each token is generated, the FSM transitions to a new state, ensuring every step remains syntactically valid according to the defined grammar.

The post-generation verification step. Even with a perfect GBNF grammar, schema validation is a critical defense-in-depth measure. It acts as a final gatekeeper, programmatically verifying that the generated data structure strictly conforms to the expected schema. This catches any edge cases where the grammar might be syntactically correct but semantically invalid, ensuring absolute data integrity before the output is consumed by downstream systems.