Function Calling

The core mechanism that forces the model to output valid, parseable JSON matching a predefined schema. This is achieved through grammar-constrained decoding or guided generation, where a Finite State Machine (FSM) tracks the current valid state and masks invalid tokens during sampling.

• Uses JSON Schema or Pydantic models as the source of truth
• Guarantees syntactic validity, eliminating brittle regex parsing
• Prevents hallucinations in structural elements like keys and data types
• Enables deterministic output when combined with temperature zero

The model intelligently selects the appropriate function from a provided set of tool definitions based on user intent. Each tool is described with a name, description, and parameters schema, allowing the model to reason about which API to call.

• Outputs a structured object with name and arguments fields
• Supports parallel tool calls for multi-step operations
• Enables the ReAct Agent Format by interleaving reasoning and action
• Integrates with the Model Context Protocol (MCP) for standardized tool interfaces

Arguments are extracted from natural language and coerced into the correct data types defined by the schema. This eliminates the ambiguity of string parsing and ensures downstream APIs receive properly typed inputs.

• Supports complex nested objects, arrays, and enums
• Handles slot filling for incomplete user requests
• Validates against data contracts before execution
• Prevents injection attacks by strictly typing all parameters

By constraining the output space to a finite set of valid tokens, function calling dramatically reduces factual hallucinations. The model cannot invent function names or parameters that don't exist in the provided schema.

• Token masking physically prevents invalid generations
• Logit bias suppresses spurious function calls
• Schema validation catches semantic errors post-generation
• Works synergistically with Retrieval-Augmented Generation for grounded arguments

Function calling maintains context across conversational turns, allowing the model to handle multi-step workflows that require chaining multiple API calls. The model remembers previous function results and uses them to inform subsequent actions.

• Manages conversational context for follow-up clarifications
• Supports multi-hop reasoning across tool calls
• Handles error recovery by requesting missing arguments
• Integrates with agentic memory for long-running sessions

Modern function calling implementations support streaming the structured output token-by-token, enabling real-time UI updates and progressive rendering of tool results before the full JSON is complete.

• Server-Sent Events (SSE) for incremental JSON delivery
• Partial parsing of incomplete JSON objects
• Enables optimistic UI updates while awaiting full execution
• Reduces perceived latency for end-users

A technique that constrains the token generation process to adhere to a predefined grammar or schema, guaranteeing syntactically valid output.

• Token Masking: Sets the probability of invalid tokens to zero during sampling.
• Finite State Machine (FSM): Tracks valid next tokens based on current grammatical state.
• Zero Retries: Eliminates the need for regeneration loops due to malformed JSON.

Techniques to reduce factually incorrect outputs. Structured output formatting is a primary defense, constraining responses to verifiable schemas.

• Schema as Guardrail: Prevents the model from inventing new fields or data types.
• Citation Forcing: Requires the model to output source pointers in a structured format.
• Deterministic Logic: Combining Temperature Zero with strict schemas eliminates syntactic creativity.