Memory-Mapped File

A memory-mapped file is a virtual memory region whose pages are backed by a file on disk, not by swap space. The operating system's virtual memory manager handles the mapping, creating a direct correspondence between file offsets and memory addresses. This abstraction allows a process to access file data using standard memory operations (load/store instructions) instead of explicit read() or write() system calls. The OS transparently handles page faults to load required data from disk into physical RAM and flushes dirty pages back to disk.

Memory mapping enables zero-copy I/O by eliminating the need to copy data from kernel buffers into user-space buffers. This dramatically reduces CPU overhead and context switches for large, sequential file operations. Performance benefits are most significant for:

• Random access patterns within large files.
• Shared read-only data between multiple processes.
• Large file processing where the working set exceeds available physical memory, leveraging the OS's efficient paging. Trade-offs exist for small, write-heavy operations where the overhead of managing memory maps and TLB (Translation Lookaside Buffer) misses can outweigh benefits.

When multiple processes map the same file region with shared mapping (MAP_SHARED on Unix, PAGE_READWRITE and FILE_MAP_WRITE on Windows), it becomes a powerful Inter-Process Communication (IPC) mechanism. Writes by one process are visible to others mapping the same region, providing high-bandwidth, low-latency data sharing. This is foundational for:

• Database systems (e.g., shared buffer pools).
• Producer-consumer workloads.
• Shared caches and in-memory data grids. Synchronization (e.g., using mutexes or semaphores) is required to prevent race conditions, as the mapping itself does not provide atomicity.

The OS uses demand paging to load file data into physical RAM only when a process accesses a memory page within the mapped region, triggering a page fault. This lazy loading is efficient for sparse access across large files, as only the touched portions consume physical memory. The mapping can be larger than available RAM; the OS's paging algorithm will swap out least-recently-used pages. Key controls include:

• madvise() (Unix) / PrefetchVirtualMemory() (Windows): Hints to the OS about access patterns (sequential, random, will-need).
• Mapping flags: MAP_POPULATE (Linux) to pre-fault and pre-load all pages eagerly.

Ensuring data is safely persisted requires understanding synchronization points. The OS periodically flushes dirty pages to disk, but explicit control is needed for durability:

• msync() (Unix) / FlushViewOfFile() (Windows): Forces the OS to write dirty pages for a specific range to disk.
• Write Ordering: msync() with MS_SYNC ensures the call blocks until writes hit durable storage.
• No Atomicity: A crash during a write can leave the file in a partially updated state. Write-ahead logging (WAL) or copy-on-write techniques are often layered on top for transactional safety.
• Metadata: Flushing data does not guarantee file metadata (size, modification time) is immediately updated; fsync() on the file descriptor is required for full durability.

While the core concept is universal, APIs and behaviors differ by OS:

• Unix/POSIX: Uses mmap() to create mapping, munmap() to unmap. PROT_READ, PROT_WRITE control protection. MAP_SHARED or MAP_PRIVATE (copy-on-write) define visibility.
• Windows: Uses CreateFileMapping() to create a file mapping object from a handle, then MapViewOfFile() to map a view into the process address space.
• Java: MappedByteBuffer class in java.nio package provides a cross-platform abstraction.
• .NET: MemoryMappedFile class in System.IO.MemoryMappedFiles namespace. Key differences exist in handling of sparse files, file locking interactions, and granularity of mappings (often tied to page size, e.g., 4KB).

A high-performance write-optimized data structure used in storage engines like RocksDB and Apache Cassandra. It batches writes in a memory-based memtable before flushing sorted, immutable files (SSTables) to disk, which are later merged in the background. This pattern is analogous to how some systems manage memory-mapped file updates.

• Advantage: Extremely high write throughput, ideal for append-heavy workloads like agent event logs.
• Trade-off: Read operations may need to check multiple SSTable levels.
• Relation to MMAP: Both optimize for sequential I/O patterns, though LSM-trees manage the merge/compaction process explicitly.

A fundamental data integrity protocol where all modifications are first written to a persistent, append-only log file before being applied to the main data structures (like a B-Tree or in-memory state). This ensures durability and enables crash recovery.

• Core Principle: Redo/Undo logging for atomicity and durability (part of ACID).
• Agentic Context: Critical for persisting agent state transitions, tool call results, or memory updates reliably before they are reflected in a memory-mapped file or vector index.
• Example: PostgreSQL, SQLite, and many modern databases use WAL as their primary durability mechanism.

The process of translating a data structure or object state into a storable or transmittable format (a byte stream). For agentic systems, efficient serialization is crucial for checkpointing state, transferring memories between processes, or writing to memory-mapped files.

• Formats: Range from language-specific (Python pickle) to language-neutral, schema-driven formats like Protocol Buffers (Protobuf), Apache Avro, or MessagePack.
• Requirements: Speed, compactness (low serialized size), and forward/backward compatibility (schema evolution).
• Use Case: Serializing an agent's working memory (a complex object graph) to a byte array before writing it to a memory-mapped region for persistence.