Occupancy

Each thread in a warp requires a dedicated set of registers for temporary data storage. The total number of registers available per Stream Multiprocessor (SM) is a fixed hardware limit. If a kernel uses many registers per thread, fewer threads can be resident concurrently, reducing occupancy. Compiler optimizations and manual register spilling are used to manage this pressure.

• Key Constraint: Register count per thread.
• Impact: High register usage directly caps the maximum number of concurrent warps.

Shared memory is a fast, programmer-managed cache shared by all threads in a thread block. Its size is limited per SM (e.g., 64KB or 96KB). The amount of shared memory requested per thread block determines how many blocks can be scheduled on an SM simultaneously. Kernels with large shared memory buffers will see lower occupancy.

• Key Constraint: Shared memory per thread block.
• Optimization: Tiling algorithms must balance block size against shared memory usage.

The dimensions of a thread block (number of threads in X, Y, Z) must align with hardware limits and workload structure. The SM has a maximum number of threads and blocks it can host. Poorly sized blocks (e.g., too few threads) can underutilize the SM, while overly large blocks may hit other resource limits first.

• Key Factors: Threads per block, blocks per SM.
• Goal: Maximize active warps within resource constraints.

GPUs execute threads in groups called warps (typically 32 threads). Control flow divergence (e.g., different if/else paths within a warp) causes serialized execution, reducing effective throughput. While divergence doesn't directly reduce the number of resident warps (occupancy), it drastically lowers the efficiency of those warps, mimicking the effect of low occupancy.

• Key Concept: SIMT (Single Instruction, Multiple Threads) execution.
• Best Practice: Minimize warp divergence for coherent execution.

This is the absolute hardware ceiling for occupancy. Each GPU architecture defines the maximum number of warps and thread blocks that can be resident on an SM. Even if register and shared memory usage are low, occupancy cannot exceed this architectural maximum. This limit is fixed and serves as the target for optimization.

• Example: An architecture may support a maximum of 64 warps per SM.
• Metric: Occupancy = (Active Warps / Maximum Warps) * 100%.

High occupancy is a means to an end: hiding latency. When a warp stalls on a long-latency operation (e.g., global memory access), the scheduler switches to a ready warp. With sufficient occupancy, the SM always has work to do, keeping execution units busy. The required occupancy depends on the kernel's instruction mix and its associated latencies.

• Core Principle: Occupancy enables latency hiding.
• Trade-off: Beyond a certain point, higher occupancy may not yield further performance gains if the kernel is compute-bound.

The Stream Multiprocessor (SM) is the fundamental, programmable computing core within a GPU architecture (or analogous unit in an NPU). It is the hardware entity where warps of threads are scheduled and executed. Occupancy is measured per SM, representing the ratio of active warps to the SM's maximum theoretical capacity. Key resources that limit occupancy, such as registers and shared memory, are allocated and managed at the SM level.

Warp Scheduling is the hardware mechanism that selects which ready warp of threads is issued to the execution units of a Stream Multiprocessor (SM). Its primary goal is to hide instruction and memory latency by keeping the cores busy. High occupancy provides the scheduler with a larger pool of eligible warps to choose from, increasing the likelihood that it can find a warp ready to execute while others are stalled, thereby improving overall hardware utilization and throughput.

SIMT is the execution model used by modern GPUs and many NPUs. A single instruction is issued to a warp (typically 32 threads), and each thread executes that instruction on its own private data. This model is fundamental to understanding occupancy:

• Threads within a warp execute in lockstep; control flow divergence (e.g., if/else) forces serialized execution, reducing effective occupancy.
• Occupancy measures how many of these SIMT warps are concurrently active on an SM, directly impacting the hardware's ability to amortize instruction fetch and decode costs across many threads.

A Thread Block is a logical grouping of threads that are guaranteed to be scheduled together on a single Stream Multiprocessor (SM). Threads within a block can cooperate via fast shared memory and synchronize using barriers. The size and resource requirements (registers, shared memory) of a thread block are the primary factors determining occupancy. A compiler or programmer must balance block size against resource limits to maximize the number of concurrent blocks resident on an SM.

Latency Hiding is the overarching goal achieved through high occupancy. When a warp stalls—for example, waiting for a high-latency global memory access—the hardware scheduler can immediately switch to executing instructions from another resident warp that is ready. This switching has zero overhead because warp state is held in hardware registers. Therefore, occupancy provides the "pool" of available warps necessary to keep the execution units perpetually busy, effectively hiding the latency of memory operations and other stalls.

Register Spilling occurs when a kernel's register usage per thread exceeds the physical register file capacity of the Stream Multiprocessor (SM). To compensate, the compiler moves excess variable data from fast registers to much slower local memory (which resides in global DRAM). This drastically increases memory latency for those variables. While spilling allows the kernel to run, it often reduces occupancy (as fewer threads can be resident due to high per-thread register demand) and introduces performance penalties, creating a direct trade-off with occupancy optimization.