Token Budget Enforcement Prompt for Chat History

This prompt is for AI engineers and platform developers who need a deterministic, model-driven method to enforce a strict token budget on conversation history before it is passed into a subsequent LLM call. The core job-to-be-done is not summarization, but prioritized selection and compression under a hard limit. You use this when your application's context window is a fixed-cost resource and you must guarantee that the assembled prompt—including system instructions, tools, and retrieved documents—does not exceed a defined maximum. The ideal user is someone who has already instrumented their application with a tokenizer and can provide an accurate token count for each candidate turn, tool output, or message block.

This prompt is designed for a specific architectural position: it sits between your session state store and your model inference call. It receives a list of pre-tokenized items, each with a token_count and a salience score, and outputs a strict subset that fits within [TOKEN_BUDGET]. It is not a general-purpose summarizer. Do not use this prompt if you need semantic compression of individual turns (use a Conversation History Compression Prompt instead) or if you need to decide when to summarize (use a Session Summarization Trigger Decision Prompt). This prompt assumes the decision to prune has already been made and the only remaining task is the selection algorithm. It is also inappropriate for low-latency, real-time audio streams where token counting on partial utterances is unreliable.

Before implementing this prompt, ensure your upstream pipeline produces reliable salience scores. The prompt's effectiveness is entirely dependent on the quality of those scores. If salience is assigned by a heuristic (e.g., recency bias) rather than a model, you may see poor retention of critical context. In high-stakes domains like healthcare or legal review, the output of this prompt must be treated as a non-deterministic compression that requires a human review step or a secondary faithfulness verification prompt before it is used as the sole context for a binding decision. The next section provides the copy-ready template you can adapt for your tokenizer and budget.

This template is the core instruction set for a token budget enforcement prompt. It instructs the model to act as a context window manager, selecting and prioritizing the most salient turns, instructions, and tool outputs to fit within a specified token limit. The prompt is designed to be dropped into a larger prompt assembly pipeline where the chat history, system prompt, and other context blocks are provided as variables. Use this template when you need a deterministic, explainable pruning step before sending a request to a model with a strict context window limit.

textCopy
You are a context window manager. Your task is to select and organize the most critical information from the provided conversation history to fit within a strict token budget.

## INPUT
- **Conversation History:** A list of turns, each with a role, content, and optional metadata.
  [CHAT_HISTORY]
- **System Instructions:** The non-negotiable instructions that define the assistant's behavior.
  [SYSTEM_PROMPT]
- **Active Tool Outputs:** Recent data returned from function calls or retrievals.
  [TOOL_OUTPUTS]

## CONSTRAINTS
- **Token Budget:** The final assembled context must not exceed [MAX_TOKENS] tokens, as counted by the [TOKENIZER_MODEL] tokenizer.
- **Priority Order:**
  1.  **System Instructions:** Must be included in full. If the system prompt alone exceeds the budget, stop and return a critical error.
  2.  **Active Tool Outputs:** Include the most recent tool output in full if it is referenced in the last user turn. Otherwise, summarize it.
  3.  **Conversation History:** Select turns based on the following salience criteria:
      - **Decisions & Actions:** Turns where a decision was made or an action was confirmed.
      - **Unresolved Questions:** The most recent open question from the user.
      - **Key Facts:** User-provided data or constraints that are still active.
      - **Corrections:** The most recent correction if it overrides a prior statement.
      - **Recency:** Prioritize more recent turns over older ones when salience is equal.
  4.  **Discard:** Remove greetings, small talk, redundant confirmations, and turns whose information has been superseded.

## OUTPUT SCHEMA
Return a valid JSON object with the following structure:
{
  "status": "success" | "error",
  "error_message": "A string explaining why the budget cannot be met, only if status is error.",
  "token_count": <integer, the final token count of the assembled context>,
  "assembled_context": [
    {
      "role": "system" | "user" | "assistant" | "tool",
      "content": "The text content of the block.",
      "priority": "critical" | "high" | "medium",
      "original_index": <integer or null, the index from the original history>
    }
  ]
}

## PROCESS
1.  Tokenize the System Instructions. If the count exceeds [MAX_TOKENS], return an error immediately.
2.  Process the Active Tool Outputs. Include or summarize based on the constraint.
3.  Score each turn in the Conversation History based on the salience criteria.
4.  Iteratively add the highest-scoring turns to the assembled context until adding the next turn would exceed the budget.
5.  Calculate the final token count of the assembled context.
6.  Return the JSON output.

To adapt this template, replace the square-bracket placeholders with your actual data. [CHAT_HISTORY] should be a serialized list of conversation turns, typically as a JSON array. [MAX_TOKENS] is the integer limit for your target model's context window minus a safety margin for the model's response. [TOKENIZER_MODEL] should be the name of the tokenizer you are using for counting (e.g., cl100k_base for GPT-4). In a production system, the token counting step in the 'PROCESS' section is often performed by your application code before and after the prompt is executed, with the prompt itself acting as the decision-making orchestrator. For high-stakes applications, always log the assembled_context and the token_count for observability and debugging.

This prompt is designed to sit inside a context window management pipeline, not as a standalone chat instruction. In production, it should be invoked programmatically before each model call when the accumulated conversation history, tool outputs, and system instructions risk exceeding the target model's context limit. The harness is responsible for assembling the full candidate payload, invoking this prompt with a strict token budget, and then passing only the validated, budget-compliant output to the downstream model. Do not rely on the model to self-enforce the budget without a post-generation validation step—models can miscount tokens or produce outputs that exceed the limit.

The implementation flow should follow a strict sequence: (1) Serialize the full candidate context including system prompt, message history, tool outputs, and any retrieved evidence into a single string. (2) Count the current token usage using the target model's tokenizer (e.g., tiktoken for OpenAI models, or the model's native token counting endpoint). (3) If usage is under budget, skip enforcement. (4) If over budget, construct the enforcement prompt by injecting the serialized context into the [FULL_CONTEXT] placeholder and the target token count into [TOKEN_BUDGET]. (5) Call a fast, cost-effective model for this compression task—GPT-4o-mini or Claude Haiku are appropriate choices since the compression model does not need to be the same as the downstream model. (6) Parse the output, which should be a JSON object with a compressed_context string and a discarded_turns array for auditability. (7) Validate the output: re-tokenize the compressed_context using the target model's tokenizer and confirm it is at or below [TOKEN_BUDGET]. If validation fails, retry once with the error message appended to the prompt. If it fails again, escalate to a fallback strategy such as aggressive truncation of the oldest turns or raising an alert for manual review.

Logging and observability are critical for this component. Every invocation should log: the pre-compression token count, the requested budget, the post-compression token count, the compression ratio, the list of discarded turn IDs, and whether validation passed on the first attempt. This data enables cost tracking, latency monitoring, and debugging when the downstream model produces poor responses due to missing context. For high-stakes applications, implement a human review queue for sessions where the compression ratio exceeds a threshold (e.g., more than 60% of turns discarded) or where the discarded turns include tool outputs with side effects. The prompt's [PRIORITY_RULES] placeholder should be populated from your application's configuration, not hardcoded, allowing different sessions or user tiers to have different retention priorities. Avoid wiring this prompt directly into the hot path without caching: if the same context state is used across multiple model calls, cache the compressed result and invalidate it only when new turns or tool outputs are added.