Slurm

Slurm's core function is managing computational jobs through sophisticated scheduling algorithms. It maintains queues (partitions) where jobs wait for resources. Key mechanisms include:

• Backfill Scheduling: Fills idle resources with smaller, lower-priority jobs without delaying high-priority work.
• Fair-Share Scheduling: Allocates resources based on historical usage to ensure equitable access among users and projects.
• Preemption: Allows higher-priority jobs to temporarily suspend or terminate lower-priority jobs, with optional checkpoint/restart.
• Job Arrays: Submits and manages collections of similar jobs (e.g., parameter sweeps) as a single entity.

Slurm abstracts cluster hardware into allocatable resources tracked in real-time. It manages:

• Nodes: Physical or virtual machines with attributes like CPU count, memory, and GPU type.
• Sockets, Cores, and Threads: Granular CPU resource allocation.
• Generic Resources (GRES): Specialized hardware like GPUs, FPGAs, or high-speed network interfaces.
• Licenses: Software license tokens as a consumable resource. The slurmctld daemon tracks resource state (idle, allocated, down, drained) and enforces allocation policies.

Slurm provides native, optimized support for launching and managing parallel jobs using the Message Passing Interface (MPI). It handles:

• Process Distribution: Maps MPI ranks to specific CPUs across allocated nodes for optimal locality.
• Environment Setup: Automatically sets MPI_HOSTFILE and other environment variables (e.g., SLURM_NODELIST, SLURM_PROCID).
• Integrated Launchers: Includes srun for direct parallel job launch and step management, eliminating the need for external tools like mpirun.
• Prolog/Epilog Scripts: Executes user-defined scripts on nodes before and after job steps for custom setup/cleanup.

Slurm maintains a detailed record of resource consumption for charging and planning. The slurmdbd (database daemon) stores:

• Job Records: User, account, resources used, walltime, and exit code.
• Association Hierarchies: Defines resource limits (FairShare, QOS, GrpTRESMins) for users within accounts.
• Quality of Service (QOS): Attributes like priority boost, preemption settings, and wallclock limits attached to jobs. This data drives the fair-share algorithm, which decays historical usage over time to prevent any single user from monopolizing resources.

For performance-critical applications, Slurm can schedule jobs based on the physical network and hardware layout of the cluster.

• Node Distance: Consults a topology plugin to understand interconnect distances (e.g., NUMA nodes, switch hops).
• Affinity and Binding: Binds tasks to specific cores, sockets, or GPUs to minimize latency and maximize cache locality.
• Consumable Resources: Tracks and allocates resources like memory bandwidth or network links that are shared among jobs on the same node. This minimizes communication overhead for tightly-coupled parallel applications like computational fluid dynamics.

Slurm is designed for production reliability in large-scale environments.

• Controller Failover: A backup slurmctld can take over if the primary fails, with state synchronization.
• Job Checkpointing: Integrates with plugins (e.g., BLCR, DMTCP) to save job state to disk for later restart, crucial for long-running jobs and preemption.
• Node Health Checks: Daemons (slurmd) monitor node health; unresponsive nodes are marked DOWN and jobs are requeued.
• State Save & Recovery: Critical state is periodically written to disk, allowing the controller to recover after a crash.

Slurm is the dominant workload manager for the world's largest academic and government research facilities. It orchestrates simulations for climate modeling, astrophysics, genomics, and materials science on machines with tens of thousands of CPU cores.

• TOP500 Supercomputers: A majority of the world's top supercomputers, including systems at Oak Ridge National Laboratory (Frontier) and Lawrence Livermore National Laboratory (Sierra), use Slurm.
• Shared University Clusters: University research computing centers rely on Slurm to fairly allocate resources among hundreds of research groups, managing complex job queues and priorities.

A compute cluster is a set of tightly or loosely connected computers, known as nodes, that work together as a single system. Slurm manages the resources of this cluster.

• Head/Login Node: The entry point where users submit jobs.
• Compute Nodes: Worker machines that execute the jobs. Slurm allocates these based on job requirements.
• Storage Nodes: Host shared filesystems accessible to all nodes.
• Management Network: A dedicated network for cluster administration and job scheduling traffic.
• High-Speed Fabric: A low-latency network (e.g., InfiniBand) for MPI communication between nodes during parallel execution.

A job scheduler is a software component that manages and allocates computational resources within a cluster. Slurm is a premier example.

• Core Functions:
  • Queuing: Holds submitted jobs in priority queues.
  • Scheduling: Uses algorithms (e.g., backfill) to decide which job runs next on available resources.
  • Dispatching: Starts the job on the allocated nodes.
  • Monitoring: Tracks job progress, resource usage, and completion status.
• Job Script: Users define resource needs (CPUs, memory, time, GPUs) and execution commands in a batch script submitted to the scheduler.

A parallel file system is a storage architecture that allows multiple compute nodes to simultaneously read from and write to a shared storage pool, providing high aggregate bandwidth essential for data-intensive HPC jobs managed by Slurm.

• Purpose: Avoids I/O bottlenecks when thousands of processes (e.g., in a simulation) need to access checkpoint files or training data concurrently.
• Examples: Lustre, BeeGFS, IBM Spectrum Scale.
• Integration: Slurm jobs are configured to access these filesystems, which are mounted across all cluster nodes. Slurm may interact with the filesystem for data-aware scheduling or burst buffer management for intermediate job storage.

InfiniBand is a high-performance network fabric, and Remote Direct Memory Access (RDMA) is its key technology. They are critical for low-latency communication in Slurm-managed HPC clusters running parallel simulations.

• InfiniBand: Provides extremely high throughput and very low latency between nodes, essential for MPI traffic in large-scale parallel codes.
• RDMA: Allows data to be transferred directly from the memory of one compute node to another without involving the CPU or OS. This drastically reduces latency and CPU overhead.
• Slurm's Role: Slurm allocates nodes with awareness of the network topology to place tightly-coupled jobs on nodes with optimal interconnect links, maximizing application performance.