Stop Sequence

Stop sequences serve as explicit termination signals that prevent models from generating beyond a defined structural boundary. When the model emits the stop token, generation halts immediately.

• Common stop sequences include \n\n, ###, </output>, or custom delimiters
• Essential for JSON mode where a closing brace } or bracket ] must be the final character
• Prevents the model from appending conversational filler like 'I hope this helps!' after structured data
• Works in conjunction with token masking to physically block continuation tokens

Stop sequences operate at the token level, not the character level. The model's tokenizer must recognize the stop sequence as a distinct token or token sequence to trigger termination.

• A stop sequence like \n\n may be a single token or two consecutive newline tokens depending on the tokenizer
• Logit bias can be applied to suppress tokens that would extend generation beyond the stop point
• In guided decoding, the finite state machine transitions to a terminal state when the stop sequence is emitted
• Mismatched tokenization can cause stop sequences to fail if the model never generates the exact token pattern

Production systems typically define multiple stop sequences to handle different termination scenarios within a single generation call.

• A JSON generation might stop on } for normal completion, \n\n for error truncation, and a custom [ERROR] token
• Stop sequences are evaluated incrementally as each token is generated, not retroactively on the full output
• Frameworks like Outlines and LMQL allow defining stop conditions as part of the grammar specification
• Multiple stop sequences prevent the model from getting stuck in repetitive loops by providing multiple escape paths

In chain-of-thought structuring, stop sequences delineate the boundary between reasoning traces and final answers, ensuring the model doesn't blend the two.

• A stop sequence like [ANSWER] signals the transition from internal reasoning to user-facing output
• The ReAct agent format uses Observation: as both a structural delimiter and an implicit stop for the previous section
• Without proper stop sequences, reasoning traces can leak into final outputs, creating confusing responses
• Stop sequences enable structured data extraction by cleanly separating the model's scratchpad from its structured payload

Stop sequences act as a critical safety mechanism against runaway generation that consumes compute resources and produces garbage output.

• Combined with a max_tokens parameter as a hard upper bound for defense-in-depth
• Models without stop sequences can enter repetitive loops, generating the same phrase indefinitely
• Grammar-constrained generation uses stop sequences as terminal symbols in the formal grammar definition
• In API deployments, stop sequences prevent billing overages from excessively long generations

When generating structured outputs against a JSON Schema, stop sequences must align with the schema's expected closing delimiters to produce valid, parseable results.

• A schema defining an array of objects expects ] as the natural stop point after the final element
• Schema validation fails if generation stops prematurely before required fields are emitted
• Pydantic and Instructor libraries implicitly manage stop sequences based on the target model's structure
• Schema drift can occur if stop sequences are hardcoded while the expected output format evolves