Dynamic Context

This method uses the model's own attention mechanisms or learned classifiers to dynamically weight or filter context. Key approaches include:

• Relevance Scoring: A lightweight classifier or cross-attention scores assess the relevance of each context chunk to the current query. Low-scoring chunks are deprioritized or removed.
• Soft Gating: Implemented via mixture-of-experts-style architectures, where a gating network decides how much to "attend to" different pieces of context, effectively blending them dynamically.
• Token-Level Pruning: Advanced systems can prune individual tokens within the context that receive minimal attention, performing lossy compression at a granular level based on the model's own focus.

For processing infinite data streams, a sliding window maintains a fixed-size cache of the most recent tokens. The StreamingLLM framework identified that stable generation requires retaining a few initial tokens as attention sinks—anchor points that absorb residual attention. The dynamic context is thus composed of:

• Attention Sink Tokens: The first 1-4 tokens of the stream, always kept in cache.
• Recent Token Window: A FIFO (First-In-First-Out) buffer of the latest N tokens.
• Eviction Policy: As new tokens arrive, the oldest tokens outside the sink and window are evicted. This provides a constant-memory, dynamic view of an arbitrarily long conversation or document.

Here, a smaller/faster compressor LLM is tasked with rewriting or distilling the raw context into a more token-efficient form before passing it to the primary reasoner LLM. Techniques include:

• Instruction-Based Compression: "Summarize this dialogue history into a concise paragraph focusing on unresolved user requests."
• Extractive to Abstractive: Converting lists of facts into a coherent narrative.
• Lossless Compression via Instructions: Using prompts like "Convert the following JSON into a minimal, single-line string without spaces." This dynamic re-encoding can drastically reduce token count while attempting to preserve informational fidelity, tailored to the needs of the downstream task.

This architectural technique uses structured metadata to dynamically assemble context. Instead of embedding full text, the system indexes chunks with attributes like topic, entity, timestamp, source, and priority. At runtime:

• Query Analyzer: Parses the user request to extract target metadata filters (e.g., time > last_week, topic == 'billing').
• Graph Traversal: If using a knowledge graph, it traverses relationships to find connected, relevant concepts.
• Hybrid Search: Combines vector similarity with metadata filtering to retrieve precisely the context that matches both semantic meaning and logical constraints. The composition of the context window is therefore a dynamic query result.

Context compression is a category of algorithms designed to reduce the token count of input context while preserving semantic utility. Key techniques include:

• Context Summarization: Using an LLM to generate a concise abstract.
• Selective Filtering: Removing tokens deemed irrelevant by a scoring model.
• Distillation: Extracting only the most salient facts or entities. These methods enable dynamic context by making room for new information.

Context retrieval is the process of fetching the most relevant information from a large corpus (e.g., a vector database) based on a query, to inject into the model's context window. It is the primary method for implementing dynamic context in Retrieval-Augmented Generation (RAG) systems, where context is not static but dynamically selected per query.

A context eviction policy is a rule set that determines which pieces of cached context are removed when memory is full. Common policies in dynamic systems include:

• Least Recently Used (LRU): Discards the oldest accessed context.
• First-In-First-Out (FIFO): Discards the earliest added context.
• Score-Based: Evicts context with the lowest relevance score. This policy directly controls the "dynamic" update of the working window.

Multi-turn context is the accumulated sequence of dialogue turns (user inputs, assistant responses, system instructions) in a conversational session. Dynamic context management is essential here to maintain coherence across long conversations without hitting the token limit. Strategies include summarizing past turns or selectively retaining only the most recent and relevant dialogue history.