Lustre

Lustre separates file metadata from file data, enabling independent scalability. Metadata Servers (MDS) manage namespace and file attributes, while Object Storage Servers (OSS) handle bulk data I/O. This decoupling prevents metadata bottlenecks when thousands of GPU nodes simultaneously access a single dataset.

• Metadata Targets (MDT) store inode and directory information
• Object Storage Targets (OST) store file data striped across multiple servers
• Clients communicate directly with OSS nodes, bypassing the MDS for data transfers

Files are divided into stripes distributed across multiple OSTs, enabling concurrent read/write operations from many clients. A single large file can be segmented across dozens of storage targets, multiplying aggregate throughput.

• Stripe count defines how many OSTs a file spans
• Stripe size controls the granularity of each segment (typically 1 MB)
• Progressive file layouts allow different stripe patterns for different regions of a file
• Ideal for large sequential I/O patterns common in training data loaders

Lustre presents a standard POSIX-compliant filesystem interface, allowing unmodified AI frameworks and data loaders to access training data without custom APIs. Client-side caching with distributed lock management reduces network round-trips for metadata operations.

• Full read/write/append semantics
• Byte-range locking for coordinated parallel writes
• LDLM (Lustre Distributed Lock Manager) ensures cache coherence across thousands of clients
• Direct I/O bypasses client cache for large sequential reads, reducing memory pressure on GPU nodes

Lustre natively supports Remote Direct Memory Access (RDMA) over InfiniBand and RoCE fabrics, enabling zero-copy data transfers directly into GPU-attached memory. The LNet networking abstraction layer provides a unified interface across multiple interconnect types.

• InfiniBand verbs for sub-microsecond latency
• RoCE v2 for converged Ethernet fabrics
• Multi-rail support for link aggregation and failover
• GPUDirect Storage integration allows direct data paths from OSTs to GPU memory, bypassing CPU buffers entirely

Lustre integrates with HSM (Hierarchical Storage Management) systems to automatically migrate cold data between high-performance flash tiers and cost-effective archive storage. Policies can be defined to move checkpoint files or aged training data to tape or object storage.

• Transparent file migration with stub files on the primary filesystem
• Policy-driven data placement based on file age, size, or access patterns
• Integration with Robinhood policy engine for automated tiering
• Reduces total cost of ownership for multi-petabyte AI datasets

Progressive File Layouts (PFL) allow different regions of a single file to use different striping configurations. This is critical for AI workloads where checkpoint files grow over time and benefit from wider striping as they expand.

• Dynamically adjusts stripe count as file size increases
• Composite layouts combine multiple layout components
• Self-extending layouts automatically apply new striping rules at size thresholds
• Eliminates manual tuning of striping parameters for evolving datasets