Memory Compression

Quantization is a lossy compression technique that reduces the numerical precision of a model's parameters (e.g., from 32-bit floating-point to 8-bit integers). This drastically reduces the memory footprint and can accelerate inference.

• Post-Training Quantization (PTQ): Applied after training; fast but may lose accuracy.
• Quantization-Aware Training (QAT): Model is trained with simulated quantization, preserving accuracy better.
• Impact: Can reduce model size by 4x (float32 to int8) with minimal performance loss, enabling larger models in constrained edge AI or on-device memory.

Pruning removes redundant or less important parameters (weights, neurons) from a neural network, creating a sparse model. Structured pruning removes entire channels or layers for efficient hardware execution.

Knowledge Distillation compresses a large, accurate teacher model into a smaller student model by training the student to mimic the teacher's outputs and internal representations.

• Result: Significantly smaller, faster models for deployment.
• Agentic Use: Enables compact small language models (SLMs) for specialized agent skills within a hierarchical memory system.

Product Quantization (PQ) and Optimized Product Quantization (OPQ) are lossy methods to compress high-dimensional vector embeddings used in vector memory stores. They split vectors into subvectors, quantize each sub-space, and represent the original vector by a short code of centroid indices.

• Memory Savings: Can reduce vector storage by 95%+ (e.g., 1024-dim float32 → 64 bytes).
• Trade-off: Enables billion-scale vector search in RAM but adds approximation error to similarity search.
• System Use: Critical for scaling semantic search in agentic long-term memory.

Deduplication identifies and stores only unique instances of identical data blocks, replacing duplicates with references. Delta encoding stores only the differences (deltas) between sequential versions of data.

• Application in Agentic Systems:
  • Deduplication: For repeated knowledge base entries or common prompt templates in memory.
  • Delta Encoding: For compressing sequential agent state updates, episodic memory logs, or versioned knowledge graph changes, where consecutive states are highly similar.
• Efficiency: Highly effective for temporal data with incremental updates.

In agentic architectures, this is a short-term, high-speed memory component that temporarily holds and manipulates information relevant to the immediate task. It is analogous to a CPU cache in traditional systems.

• Function: Provides fast, context-aware access to recent interactions, tool outputs, and intermediate reasoning steps.
• Compression Context: While typically small and fast, compression algorithms (especially low-latency ones like LZ4) can be used to pack more relevant context into a fixed-size buffer, effectively extending the agent's immediate recall.

A virtual memory management scheme where idle pages of memory are moved from RAM to a secondary storage area (swap space) to free physical memory for active processes. When swapped-out data is needed again, it is paged back in, often triggering disk I/O.

• Performance Impact: Excessive swapping ('thrashing') causes severe latency due to slow disk access.
• Compression Benefit: In-memory compression can be used as an alternative or complement to swapping. By compressing idle pages in RAM, the system can avoid or delay the need to swap to disk, maintaining much lower access latency.

The multi-level structure of CPU caches, where each successive level (L1, L2, L3) is larger, slower, and shared among more cores. This hierarchy is optimized for memory locality.

• L1 Cache: Smallest, fastest, per-core. Focus on ultra-low latency.
• L2/L3 Cache: Larger, slower, often shared. Focus on capacity and reducing main memory accesses.
• Compression Relevance: While not typically compressed due to extreme latency constraints, the principles of cache efficiency (exploiting spatial/temporal locality) parallel the goals of memory compression at higher levels. Research into compressed caches exists for specific workloads.

A non-volatile memory tier that retains data across system restarts, blurring the line between memory and storage. Technologies include NVMe SSDs and Intel Optane Persistent Memory.

• Characteristic: Offers higher density and persistence than DRAM, but with higher latency.
• Compression Criticality: Compression is highly valuable in this layer to maximize the utility of this dense, byte-addressable space. It reduces write amplification in SSDs and increases the effective capacity for in-memory databases or agentic long-term memory stores.