Context Window

The context window represents an absolute computational limit—not a soft guideline. For embedding models, this dictates the maximum input length for generating a single vector representation. For language models, it governs how much text can be attended to simultaneously. Exceeding this limit triggers truncation errors or requires chunking strategies.

• Embedding models: Typically 512–8192 tokens (e.g., text-embedding-3-large supports 8191)
• Language models: Range from 4K tokens (older GPT models) to 1M+ tokens (Gemini 1.5 Pro)
• Token ≠ word: A token averages ~0.75 words in English, meaning a 512-token window holds roughly 380 words

The context window directly determines the maximum chunk size in semantic indexing pipelines. Chunks must fit entirely within the embedding model's window, but optimal chunk size is often smaller to preserve semantic coherence.

• Upper bound: Chunk size ≤ embedding model's context window
• Practical sizing: 256–512 tokens for dense retrieval; 1024–2048 for generative context
• Overlap margin: Typically 10–20% of chunk size to prevent context fragmentation at boundaries
• Trade-off: Larger chunks capture more context but dilute semantic specificity in vector search

The context window size is constrained by the quadratic complexity of self-attention. Standard transformer attention requires O(n²) computation and memory relative to sequence length n, making long contexts extremely resource-intensive.

• Quadratic bottleneck: Doubling context length quadruples attention computation
• Mitigation strategies: FlashAttention, ring attention, and sparse attention patterns reduce memory footprint
• Position encoding limits: Models like RoPE (Rotary Position Embedding) must be configured for the target context length during training
• KV-cache memory: Long contexts consume substantial GPU memory for storing key-value pairs during autoregressive generation

The context window shapes retrieval strategy design. When the generative model's window is large enough to hold dozens of retrieved chunks, retrieval can prioritize high recall. When windows are constrained, precision becomes paramount.

• Large windows (128K+): Enable dumping entire documents into context, reducing reliance on precise chunking
• Small windows (4K–8K): Require aggressive chunking and re-ranking to fit only the most relevant passages
• Parent-child retrieval: Small chunks for semantic matching, larger parent documents for context delivery
• Multi-hop reasoning: Requires sufficient window space to hold intermediate retrieval results alongside the original query

In multi-turn agentic systems, the context window must be budgeted across conversation history, retrieved documents, system prompts, and the current query. Mismanagement leads to catastrophic forgetting of earlier instructions or relevant context.

• System prompt overhead: Can consume 500–2000+ tokens before any user input
• History accumulation: Each turn adds query-response pairs, rapidly consuming available space
• Sliding window strategies: Dropping oldest messages when the budget is exceeded
• Summarization compression: Periodically condensing conversation history into a summary to reclaim tokens
• Context window exhaustion: Results in models "forgetting" early instructions or retrieved facts

Different model families impose distinct context window limits that directly influence architecture decisions. Selecting a model requires matching its window capacity to the application's context demands.

• OpenAI: GPT-4o supports 128K tokens; text-embedding-3-large supports 8191 tokens
• Anthropic: Claude 3.5 Sonnet offers 200K tokens
• Google: Gemini 1.5 Pro reaches up to 1M tokens with near-perfect retrieval accuracy
• Open source: Llama 3.1 supports 128K; Mistral Large supports 128K
• Embedding-specific: intfloat/e5-mistral-7b-instruct supports 32K tokens for long-document embedding

The fundamental preprocessing step that segments raw text into atomic units called tokens. A model's context window is measured in these tokens, not raw characters or words. Different tokenizers (e.g., BPE, WordPiece) produce different token counts for identical text, directly impacting how much information fits within a given window limit.

The methodology for segmenting unstructured documents into discrete, semantically coherent pieces to fit within an embedding model's context window. Effective chunking respects the model's maximum token limit while preserving conceptual integrity. Strategies include:

• Fixed-size chunking: Splits at exact token counts with overlap
• Semantic chunking: Uses embedding similarity to find natural break points
• Recursive splitting: Applies a hierarchy of separators (paragraph → sentence → word)

A configurable number of tokens shared between adjacent chunks to prevent context fragmentation. When a fixed-size split severs a sentence mid-thought, the overlap ensures the subsequent chunk contains the preceding tokens needed for coherence. Typical overlap ranges from 10-20% of chunk size. Without overlap, critical entities and their referents may be separated across chunk boundaries, degrading retrieval quality.

The fixed length of the output vector generated by an embedding model. Common dimensions include 768 (BERT-base), 1024 (BGE-large), and 1536 (OpenAI ada-002). Higher dimensions capture more nuanced semantic features but increase storage and computational load. The context window determines the maximum input length, while the embedding dimension defines the output representation size—two distinct but equally critical constraints in retrieval architecture.

The engineering discipline of maintaining session state across multi-turn interactions while respecting the model's finite context window. Techniques include:

• Sliding window: Retains only the most recent N tokens
• Summarization buffers: Compresses older turns into concise summaries
• Vector-backed memory: Stores conversation history externally and retrieves relevant segments on demand When the cumulative dialogue exceeds the context window, the system must strategically evict or compress older information without losing critical context.

A retrieval strategy that decouples the search granularity from the context window utilization. A small, targeted 'child' chunk (e.g., 256 tokens) is used for precise semantic matching during vector search. However, when retrieved, the larger surrounding 'parent' document block (e.g., 1024 tokens) is returned to the language model. This maximizes the context window's capacity for meaningful information while maintaining high retrieval precision.