Dynamic Context

This is the foundational technique for dynamic context. Instead of loading an entire corpus, a retrieval-augmented generation (RAG) system uses a query to fetch only the most relevant document chunks from a vector database. The process involves:

• Query Encoding: The user's question or the agent's current state is converted into a vector embedding.
• Similarity Search: The system performs a nearest-neighbor search (e.g., using cosine similarity) against pre-indexed document embeddings.
• Context Injection: The top-k most relevant chunks are inserted into the model's prompt. This ensures the context window contains only high-utility, task-specific information, dynamically updating with each new query.

This technique progressively condenses long context. A hierarchical summarization pipeline processes information in stages:

• Chunk-Level Summaries: Individual document segments are first summarized.
• Roll-Up Summaries: Those segment summaries are then aggregated and summarized again at a chapter or document level.
• Conversation Memory: In multi-turn dialogues, past exchanges are periodically summarized into a conversation summary that is carried forward, while raw turn history is evicted. This creates a recursive memory structure where only compressed representations of past context are retained, freeing tokens for new interaction.

This method uses the model's own attention mechanisms or learned classifiers to dynamically weight or filter context. Key approaches include:

• Relevance Scoring: A lightweight classifier or cross-attention scores assess the relevance of each context chunk to the current query. Low-scoring chunks are deprioritized or removed.
• Soft Gating: Implemented via mixture-of-experts-style architectures, where a gating network decides how much to "attend to" different pieces of context, effectively blending them dynamically.
• Token-Level Pruning: Advanced systems can prune individual tokens within the context that receive minimal attention, performing lossy compression at a granular level based on the model's own focus.

For processing infinite data streams, a sliding window maintains a fixed-size cache of the most recent tokens. The StreamingLLM framework identified that stable generation requires retaining a few initial tokens as attention sinks—anchor points that absorb residual attention. The dynamic context is thus composed of:

• Attention Sink Tokens: The first 1-4 tokens of the stream, always kept in cache.
• Recent Token Window: A FIFO (First-In-First-Out) buffer of the latest N tokens.
• Eviction Policy: As new tokens arrive, the oldest tokens outside the sink and window are evicted. This provides a constant-memory, dynamic view of an arbitrarily long conversation or document.

Here, a smaller/faster compressor LLM is tasked with rewriting or distilling the raw context into a more token-efficient form before passing it to the primary reasoner LLM. Techniques include:

• Instruction-Based Compression: "Summarize this dialogue history into a concise paragraph focusing on unresolved user requests."
• Extractive to Abstractive: Converting lists of facts into a coherent narrative.
• Lossless Compression via Instructions: Using prompts like "Convert the following JSON into a minimal, single-line string without spaces." This dynamic re-encoding can drastically reduce token count while attempting to preserve informational fidelity, tailored to the needs of the downstream task.

This architectural technique uses structured metadata to dynamically assemble context. Instead of embedding full text, the system indexes chunks with attributes like topic, entity, timestamp, source, and priority. At runtime:

• Query Analyzer: Parses the user request to extract target metadata filters (e.g., time > last_week, topic == 'billing').
• Graph Traversal: If using a knowledge graph, it traverses relationships to find connected, relevant concepts.
• Hybrid Search: Combines vector similarity with metadata filtering to retrieve precisely the context that matches both semantic meaning and logical constraints. The composition of the context window is therefore a dynamic query result.