LMQL

LMQL allows developers to express lexical constraints directly within prompts using Python-like syntax. This ensures generated text adheres to specific formats, data types, or vocabularies.

• Enforce JSON schema compliance during generation
• Restrict outputs to predefined sets of valid tokens
• Prevent hallucinations by limiting the model to factual entity lists
• Use where clauses to define stopping conditions and validity checks

Unlike static prompting, LMQL treats the generation process as a scriptable program. Developers can use loops, conditionals, and variable assignment to create dynamic, multi-step reasoning flows.

• Implement branching logic based on intermediate model outputs
• Use for loops to iterate over lists and generate structured lists
• Store intermediate results in Python variables for later use in the prompt
• Build complex ReAct-style agent loops natively

LMQL performs eager validation at each decoding step. Before a token is generated, the runtime checks it against the defined constraints and sets the probability of invalid tokens to zero.

• Guarantees syntactically valid output without post-processing
• Uses token masking to physically prevent invalid generations
• Supports regular expressions and context-free grammar constraints
• Dramatically reduces the need for retry logic and error handling

LMQL operates as a high-level abstraction layer that compiles down to various inference backends. The same query logic runs on different models without modification.

• Supports OpenAI API, Hugging Face Transformers, and llama.cpp
• Compiles constraints into backend-specific logit bias or FSM instructions
• Enables seamless switching between local and cloud-hosted models
• Standardizes structured generation across the entire model ecosystem

LMQL supports hierarchical generation where the output of one query segment becomes the input for another. This enables complex decomposition of reasoning tasks.

• Use the @ operator to call nested query functions
• Build multi-hop reasoning chains with intermediate validation
• Implement self-consistency by generating and comparing multiple samples
• Structure tree-of-thought exploration within a single script

LMQL can automatically adjust decoding parameters like temperature and beam width based on the constraint context. This removes the need for manual hyperparameter tuning.

• Automatically sets temperature to zero for deterministic constraint blocks
• Applies beam search when multiple valid outputs are possible
• Optimizes token lookahead to satisfy complex grammatical constraints
• Balances creativity and adherence within different segments of the same prompt

The foundational technique that constrains the token generation process to adhere to a predefined grammar or schema. Unlike post-processing, guided decoding modifies the logit probabilities during inference, setting invalid token probabilities to zero. This ensures syntactically valid output at every step rather than hoping the model follows instructions. LMQL compiles constraints into a token-level mask that integrates directly with this mechanism.

Forces a model's output to conform to a formal grammar, typically a Context-Free Grammar (CFG). This guarantees parseable, schema-compliant results by defining recursive production rules that govern every valid sequence. LMQL's constraint system acts as a higher-level abstraction over these grammars, allowing developers to express constraints in a Python-like syntax rather than writing raw GBNF or CFG rules.

A low-level mechanism that dynamically sets the probability of invalid tokens to zero during each decoding step. The model's output logits are modified before softmax, physically preventing the sampler from selecting out-of-schema tokens. LMQL's runtime computes these masks efficiently by evaluating which tokens satisfy the current constraint state, enabling complex multi-step constraints without performance degradation.

An abstract computational model that tracks the current valid state of an output sequence and determines the set of permissible next tokens. In guided generation, the FSM transitions between states as tokens are produced, enforcing the grammar incrementally. LMQL compiles high-level constraints into an efficient FSM representation that operates transparently during decoding.

A vocabulary for annotating and validating JSON documents, defining structure, data types, and constraints. When used with structured generation, JSON Schema serves as the target specification that the model must satisfy. LMQL can express JSON Schema constraints natively through its type system and structural constraints, ensuring generated JSON is both syntactically valid and semantically correct.