Token Limit

Strategically partitioning the finite token limit between system instructions, retrieved documents, conversation history, and model output. A common failure mode is exhausting the budget on low-relevance retrieved chunks, leaving no room for the actual reasoning step.

• System prompt: Reserve 5-10% for immutable instructions
• Retrieval window: Allocate 60-70% for grounding documents
• History buffer: Use a sliding window of recent turns
• Output reserve: Always leave headroom for the max generation length

When input exceeds the token limit, the truncation strategy determines what information is discarded. Naive right-truncation often removes critical instructions or the user's latest query.

• Head truncation: Drops the oldest tokens first, preserving recent context
• Priority-aware truncation: Assigns retention priority to system prompts and the final user turn
• Summary compression: Replaces truncated history with an LLM-generated summary to preserve state
• Chunk-level truncation: Removes entire retrieved documents rather than cutting mid-sentence

Precise token counting is essential for programmatic context management. A mismatch between the counting library and the model's actual tokenizer can cause silent truncation or rejected API calls.

• Use the model's native tokenizer (e.g., tiktoken for OpenAI models)
• Account for special tokens like BOS, EOS, and chat template delimiters
• Count tokens on the serialized chat format, not raw text
• Implement server-side validation before dispatch to prevent hard errors

When the combined prompt exceeds the limit, compression techniques reduce token count while preserving semantic fidelity. This is distinct from truncation—it aims to retain all essential information in a denser form.

• Extractive summarization: Pull key sentences from retrieved documents
• Abstractive compression: Use a smaller model to rewrite chunks concisely
• Entity extraction: Replace verbose passages with structured key-value pairs
• Instruction distillation: Condense verbose system prompts into terse, imperative directives

The token limit is a combined ceiling on both input and output. Failing to reserve sufficient tokens for the model's response causes generation to halt mid-sentence, a critical failure in user-facing applications.

• Set max_tokens explicitly rather than relying on defaults
• Calculate: max_tokens = model_limit - prompt_tokens - safety_margin
• For streaming applications, monitor token consumption in real-time
• Implement graceful fallback when the output limit is reached (e.g., offer a continuation prompt)

Token consumption directly drives API pricing and time-to-first-token. Engineering decisions around context window utilization must balance retrieval quality against operational cost.

• Prefill latency: Scales with input token count; optimize retrieved chunk count
• Decode latency: Scales with output token count; enforce concise generation
• Cost optimization: Cache frequently used system prompts via prefix caching
• Monitoring: Track tokens-per-request as a core SLO alongside latency and error rate

The maximum span of text an LLM can process in a single inference request, measured in tokens. While token limit is the hard numerical ceiling, the context window defines the model's immediate working memory. Modern models like GPT-4 Turbo offer 128K-token windows, but performance degrades predictably in the Lost-in-the-Middle zone.

• Claude 3: 200K token context window
• Gemini 1.5 Pro: Up to 1M tokens in research preview
• Key tradeoff: Larger windows increase latency quadratically due to self-attention complexity

A set of techniques that condense lengthy prompts into smaller, information-dense representations to reduce token usage without sacrificing essential semantic content. Methods include:

• LLMLingua: Uses a small language model to identify and remove redundant tokens while preserving task-critical information
• Selective Context: Drops lexical units with low self-information scores
• Gist Tokens: Trains special compressed representations that replace full prompts

Effective compression can reduce token consumption by 2–5x while maintaining downstream task accuracy within 95% of the uncompressed baseline.

A memory buffer that stores pre-computed key and value tensors from previous decoding steps, eliminating redundant computation during autoregressive generation. The KV-Cache grows linearly with sequence length and is often the primary bottleneck for long-context inference.

• GQA (Grouped-Query Attention): Reduces KV-Cache size by sharing key-value heads across multiple query heads—used in Llama 2 70B and Mistral
• Multi-Query Attention (MQA): Extreme variant with a single KV head, used in PaLM
• Prefix Caching: Reuses KV-Cache for static prompt prefixes across multiple requests

Without KV-Cache optimization, serving long-context models becomes economically infeasible due to GPU memory constraints.

The process of segmenting raw text into discrete atomic units from a fixed vocabulary, serving as the fundamental input representation for language models. Tokenization directly impacts how efficiently you use your token budget.

• Byte-Pair Encoding (BPE): Iteratively merges frequent byte pairs—used by GPT models. Common English words become single tokens; rare terms fragment into multiple subwords
• SentencePiece: Treats input as raw byte stream, language-agnostic—used by Llama and Mistral
• WordPiece: BPE variant with likelihood-based merging—used by BERT

Practical impact: A 1,000-word English article typically consumes ~1,300–1,500 tokens. Non-English languages with different morphology can consume 2–3x more tokens for equivalent semantic content.

A documented performance failure mode where information placed in the center of a long context window is significantly less likely to be retrieved or utilized compared to content at the beginning (primacy effect) or end (recency effect).

• Research from Stanford, UC Berkeley, and Samaya AI demonstrated this across multiple model architectures
• Impact on token budget: Simply cramming more data into the context window does not guarantee the model will attend to it
• Mitigation strategies:
  • Place critical instructions and data at the very beginning or end of the prompt
  • Use re-ranking to ensure the most relevant retrieved chunks occupy high-attention positions
  • Split complex tasks into multi-turn interactions rather than single massive prompts

An exact-attention algorithm that minimizes high-bandwidth memory reads and writes between GPU HBM and SRAM by using tiling and recomputation. FlashAttention dramatically speeds up attention computation and reduces memory footprint for long sequences.

• FlashAttention-2: Up to 2x faster than v1, achieves ~70% of theoretical maximum FLOP utilization on A100 GPUs
• FlashAttention-3: Optimized for H100 architectures with asynchronous processing
• Practical benefit: Makes 128K+ context windows economically viable by reducing the quadratic memory cost of attention to near-linear in practice

Without FlashAttention, processing a 100K-token sequence on a single GPU would be prohibitively slow or impossible due to out-of-memory errors.