Glossary

Chat Template

A structured formatting script within a tokenizer that converts a sequence of chat messages into a single tokenized string with the appropriate control tokens for a specific instruction-tuned model.

Get in touch Learn more

ML engineer managing model versions on laptop, version history visible, technical Git-like workflow.

INSTRUCTION FORMATTING

What is a Chat Template?

A structured formatting script within a tokenizer that converts a sequence of chat messages into a single tokenized string with the appropriate control tokens for a specific instruction-tuned model.

A chat template is a deterministic formatting script integrated into a tokenizer that transforms a structured list of chat messages—each with a role (e.g., system, user, assistant) and content—into a single, tokenized string. This process injects the model-specific special tokens and control sequences required to delineate conversational turns, ensuring the underlying language model correctly interprets the dialogue structure.

Without a chat template, a raw sequence of messages would lack the necessary syntactic boundaries, causing the model to misinterpret speaker roles or ignore system-level instructions. The template is applied during both encoding (converting messages to token IDs for inference) and decoding (parsing a generated string back into structured messages), making it a critical component for the correct operation of instruction-tuned and chat-fine-tuned models.

INSTRUCTION-TUNED FORMATTING

Key Characteristics of Chat Templates

Chat templates are the critical bridge between conversational user interfaces and the raw tokenization pipeline, ensuring that a model correctly interprets the roles and boundaries of a multi-turn dialogue.

Structured Role Delineation

A chat template's primary function is to inject control tokens that explicitly separate system instructions, user queries, and assistant responses. This prevents the model from confusing dialogue turns.

System Prompt: Encapsulated with tokens like <|system|> to set global behavior.
User Turn: Prefixed with <|user|> to indicate the external input.
Assistant Turn: Wrapped with <|assistant|> and often an end-of-turn token like <|end|> to signal the model to stop generating.

EXPLORE

Additive Generation Prompt

To trigger a response, the template appends a generation prompt at the very end of the tokenized sequence. This pre-fills the start of the assistant's turn, forcing the model to continue the pattern rather than add a new user turn.

Example: The string <|assistant|> is appended after the user's message.
Mechanism: The model sees this as a partially complete assistant turn and generates the completion, ensuring the output is a direct reply.

EXPLORE

Jinja2 Programmatic Logic

Modern chat templates are not static strings; they are executable Jinja2 scripts stored within the tokenizer configuration. This allows for complex, model-specific logic during pre-processing.

Conditional Logic: The template can loop through a list of message dictionaries and apply different formatting based on the role key.
Whitespace Control: Jinja2 strips unnecessary whitespace and newlines that could negatively impact tokenization for code-generation models.

EXPLORE

Tokenizer-Specific Serialization

The chat template converts a high-level list of message objects into a single, flat string that can be passed directly to the tokenizer's encode method. This serialization is lossless regarding dialogue structure.

Input: [{"role": "user", "content": "Hello!"}]
Output: A string like "<|user|> Hello!</s> <|assistant|> " that includes all necessary special tokens.
Integration: This happens seamlessly when using tokenizer.apply_chat_template(), abstracting away the raw string manipulation.

EXPLORE

Model-Specific Fingerprint

A chat template is a unique fingerprint of a model's instruction-tuning process. Using the wrong template breaks the model's expected input distribution, causing severe performance degradation.

LLaMA Format: Uses [INST] and [/INST] tags.
ChatML Format: Uses <|im_start|> and <|im_end|> markers.
Mistral Format: Uses a specific string pattern without explicit role tokens for the user.
Consequence: Mismatched templates lead to the model generating garbage, repeating the prompt, or failing to follow instructions.

EXPLORE

Tool-Calling Integration

Advanced chat templates extend their syntax to support function calling or tool use. They define a structured way to serialize a function definition and its arguments into the token stream.

Function Definition: A system message containing a JSON schema of the available tool is injected using a specific control token.
Function Call: The assistant's turn is formatted to output a special token like <function_call> followed by the JSON arguments.
Function Response: The result of the tool execution is injected back into the conversation history with a <function_response> token, allowing the model to continue reasoning.

EXPLORE

CHAT TEMPLATE

Frequently Asked Questions

Clear answers to common questions about how chat templates structure conversational data for instruction-tuned language models.

A chat template is a structured formatting function within a tokenizer that converts a sequence of conversational messages—each with a role (e.g., system, user, assistant) and content—into a single tokenized string suitable for a specific instruction-tuned model. It works by applying a predefined Jinja2 template that inserts the model's required control tokens and special tokens around each message. For example, a template might prepend <|im_start|>user\n before a user's message and append <|im_end|>\n after it, then concatenate all messages and apply the tokenizer's encoding logic. This ensures the model receives the conversation in the exact format it was fine-tuned on, preserving the semantic distinction between speakers and system-level instructions.

Enabling Efficiency, Speed & Accuracy

Intelligent Analysis, Decision & Execution

We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.

Talk to Us

Search across company data

Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.

Useful when people spend too long searching or get different answers from different systems.

Enterprise searchRAGPermissions

Automate internal workflows

Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.

Useful when repetitive work moves across multiple tools and teams.

AI agentsWorkflow automationGovernance

Add AI to products and internal tools

Build assistants, guided actions, or decision support into the software your team or customers already use.

Useful when AI needs to be part of the product, not a separate tool.

AI integrationDecision supportModel routing

TOKENIZATION ECOSYSTEM

Related Terms

Understanding chat templates requires familiarity with the broader tokenization pipeline and the special tokens that control model behavior.

Special Tokens

Reserved vocabulary entries that act as control characters within a chat template. They delineate conversational roles and structure.

[BOS] / <|begin_of_text|>: Marks the start of a sequence.
[EOS] / <|end_of_text|>: Marks the end of a generation turn.
[INST] / <|user|>: Delimits the start of a user instruction.
[ASST] / <|assistant|>: Delimits the start of the model's response.

A chat template's primary job is to insert these tokens in the correct order around the message content.

Tokenization Pipeline

The multi-step process that converts raw text into model inputs. A chat template is applied as a pre-processing step before the core tokenizer runs.

Step 1: The chat template formats messages into a single string with special tokens.
Step 2: Normalization standardizes the string (e.g., NFC Unicode).
Step 3: Pre-tokenization splits the string into coarse word-level units.
Step 4: The BPE or WordPiece model segments words into subword token IDs.
Step 5: Post-processing adds the final BOS/EOS tokens and generates the attention mask.

Encoding & Decoding

The two fundamental operations of any tokenizer, which a chat template orchestrates for conversational data.

Encoding: The chat template converts a list of role-dictated messages into a single string, which is then encoded into a list of token IDs via the vocabulary.
Decoding: The model's output token IDs are converted back to text. The chat template is typically not applied during decoding, but special tokens like <|end_of_turn|> signal when to stop.

A mismatch between the template used for training and inference will cause severe performance degradation.

Hugging Face Chat Templates

The chat_template attribute on Hugging Face tokenizers is a Jinja2 template that standardizes conversational formatting.

Stored directly in the tokenizer_config.json file.
Automatically applied when calling tokenizer.apply_chat_template(messages).
Handles the insertion of generation prompts (e.g., <|assistant|>) to cue the model's response.
Supports multi-turn history by concatenating previous assistant and user turns.

This system ensures that any model can be used with a uniform conversational API.

Attention Mask Generation

A binary tensor created during tokenization that tells the model which tokens to attend to. Chat templates influence mask construction for training.

Content tokens (user/assistant messages) receive a mask value of 1.
Padding tokens receive a mask value of 0.
In instruction fine-tuning, the loss mask is a related concept where the model is only trained to predict the assistant's response, not the user's prompt.

The chat template often defines which spans of the sequence are user input (masked from loss) and which are the target assistant output.

Vocabulary & Token ID Mapping

The fixed set of unique tokens a model recognizes. A chat template relies on specific entries existing in this vocabulary.

The template references special tokens by their string representation (e.g., <|im_start|>).
The tokenizer converts these strings to their unique Token ID integers.
If a required special token is missing from the vocabulary, the template will fail or produce incorrect formatting.
Vocabulary size is a fixed hyperparameter (e.g., 32,000 for Llama 2, 128,000 for GPT-4) that includes these control tokens.

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.

Limited slotsGet a Free AI Consultation

How We Work

Custom AI workflows for your Business

One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.

Talk to Us

Chat Template

What is a Chat Template?

Key Characteristics of Chat Templates

Structured Role Delineation

Additive Generation Prompt

Jinja2 Programmatic Logic

Tokenizer-Specific Serialization

Model-Specific Fingerprint

Tool-Calling Integration

Frequently Asked Questions

Intelligent Analysis, Decision & Execution

Search across company data

Automate internal workflows

Add AI to products and internal tools

Prasad Kumkar

Partnered with leading AI, data, and software stack.

Custom AI workflows for your Business

Review the use case

Pick the right approach

Build the first useful version

Improve from there