Workgroup Size

Workgroup size defines the number of threads that are bundled together for cooperative execution on a single Compute Unit (CU) or Streaming Multiprocessor (SM). This grouping is the fundamental unit of scheduling for the hardware's SIMD (Single Instruction, Multiple Data) or SIMT (Single Instruction, Multiple Thread) engine. The hardware executes threads in groups called warps (NVIDIA) or wavefronts (AMD), and the workgroup size must be a multiple of this hardware warp/wavefront size (typically 32 or 64) to avoid underutilization. The compiler and runtime map each workgroup to the hardware's execution resources, managing register allocation and shared memory per workgroup.

Occupancy is the ratio of resident workgroups to the maximum possible on a compute unit. A larger workgroup consumes more registers and shared memory, potentially reducing the number of concurrent workgroups and lowering occupancy. However, sufficient occupancy is critical for latency hiding: when some threads are stalled waiting for memory, the hardware can switch to executing instructions from other ready threads within other resident workgroups. The optimal workgroup size maximizes occupancy while providing enough threads to keep the hardware busy during memory access latencies. Profiling tools report occupancy metrics to guide this tuning.

Workgroup size and organization directly influence memory coalescing, a critical optimization for bandwidth. For global memory accesses, consecutive threads within a workgroup should access consecutive memory addresses. This pattern allows the memory controller to combine multiple accesses into a single, wide transaction. A poorly chosen workgroup size or thread indexing scheme can result in non-coalesced access, where each thread's request is issued separately, devastating effective bandwidth. The workgroup dimensions (e.g., (16, 16) for a 2D kernel) should align with the data access patterns of the algorithm to promote coalescing.

Threads within a workgroup can synchronize efficiently using barrier instructions (e.g., barrier(CLK_LOCAL_MEM_FENCE)). This allows for cooperative algorithms like parallel reductions or tiled matrix multiplication where threads load data into scratchpad memory (a.k.a. shared memory/LDS), synchronize, and then compute. The workgroup size defines the scope and size of this collaborative workspace. The amount of shared memory required per workgroup is a key constraint; exceeding the per-CU limit will prevent the workgroup from launching. Synchronization is only defined within a workgroup; communication between workgroups requires atomic operations on global memory.

Finding the optimal workgroup size is a prime target for auto-tuning due to complex interactions with hardware. The configuration space includes:

• Total size (e.g., 64, 128, 256, 512).
• Multi-dimensional layout (e.g., (8,8), (16,16), (32,4)).

Kernel tuners like CLTune or AutoKernel perform a parameter search, often guided by Bayesian optimization or genetic algorithms, to evaluate performance across this space. A performance model may predict optimal sizes based on kernel characteristics (compute vs. memory bound) and hardware limits (register count, shared memory). The search must respect hardware constraints (max workgroup size, which is often 1024 or higher).

Choosing workgroup size involves balancing competing factors:

• Large Workgroups: Improve latency hiding within the group and can reduce global synchronization overhead. Risk higher register pressure, lowering occupancy.
• Small Workgroups: Increase potential occupancy and may fit more concurrent groups. Risk insufficient parallelism to hide memory latency and increased global scheduling overhead.

Common heuristics: Start with a size that is a multiple of the warp/wavefront size (e.g., 128 or 256). For memory-bound kernels, prioritize sizes that maximize occupancy. For compute-bound kernels, ensure the size provides enough independent instructions to keep all ALUs busy. The final choice is always validated with a kernel profiler measuring execution time and hardware performance counters.

Optimal workgroup size is dictated by the underlying hardware's execution units and memory architecture.

• NVIDIA GPU (CUDA): Aim for a multiple of 32 (the warp size). Common sizes are 128, 256, or 512 threads per block to maximize occupancy and hide latency.
• AMD GPU (ROCm): Target a multiple of 64 (the wavefront size). Sizes of 64, 128, or 256 are typical.
• Mobile/Edge NPUs (e.g., Qualcomm Hexagon): Often have smaller, fixed-size SIMD lanes (e.g., 128-bit). Workgroups may be smaller, such as 64 or 128, to fit constrained shared memory.
• Apple Silicon (GPU): Uses threadgroups aligned with its SIMD width. Sizes like 128 or 256 are efficient, but performance depends heavily on memory coalescing patterns.

The nature of the kernel's bottleneck influences the ideal workgroup size.

• Memory-Bound Kernels: Performance is limited by memory bandwidth. Use larger workgroups (e.g., 256-512) to increase the number of concurrent memory requests, improving latency hiding and enabling better memory coalescing.
• Compute-Bound Kernels: Performance is limited by ALU throughput. Use smaller workgroups (e.g., 64-128) that fit more concurrent groups on a core, increasing occupancy and utilization of arithmetic units.
• Rule of Thumb: Profile with a kernel profiler to identify the bottleneck, then adjust workgroup size accordingly.

Workgroup size directly impacts the consumption of on-chip, fast memory resources.

• Shared Memory (Local Data Share): Each workgroup allocates a static block of shared memory. Larger workgroups consume more, potentially limiting the number of concurrent groups that can reside on a compute unit.
• Register Pressure: Each thread requires architectural registers. Larger workgroups increase total register demand per group, which can reduce occupancy if registers are exhausted.
• Heuristic: Use the vendor's occupancy calculator or profiler to find the maximum workgroup size before register/spill pressure causes performance degradation.

Workgroup size impacts the penalty for thread divergence.

• Within a warp (NVIDIA, 32 threads) or wavefront (AMD, 64 threads), divergent if/else or loop paths cause serialized execution.
• Smaller workgroups may increase the probability that all threads in a warp follow the same path, reducing divergence.
• Heuristic: For kernels with complex, data-dependent control flow, experiment with workgroup sizes that are exact multiples of the hardware's warp/wavefront size to pack similar threads together.

Auto-tuning is the definitive method for finding the optimal workgroup size. A typical search involves:

1. Define the Configuration Space: Parameterize workgroup size (e.g., [32, 64, 128, 256, 512]).
2. Use a Search Strategy:
   • Exhaustive Search: Test all combinations (feasible for 1-2 parameters).
   • Bayesian Optimization: Models performance to intelligently sample the space.
   • Genetic Algorithms: Evolve populations of configurations.
3. Evaluate with a Benchmark: Run the kernel with each configuration, measuring execution time or compute throughput.
4. Select the Optimum: Choose the configuration with the best performance metric.

Tools like CLTune, Kernel Tuner, or Ansor automate this process.

For a tiled matrix multiplication kernel on an NVIDIA A100 GPU:

• Key Tunable: BLOCK_SIZE (which defines the workgroup dimensions as BLOCK_SIZE x BLOCK_SIZE).
• Typical Search Space: BLOCK_SIZE ∈ [8, 16, 32, 64].
• Trade-offs:
  • BLOCK_SIZE=16: 256 threads/block. Good occupancy, but may not fully utilize shared memory bandwidth.
  • BLOCK_SIZE=32: 1024 threads/block (max). Maximizes parallelism per block but increases register pressure, potentially lowering occupancy.
• Optimal Result: For large matrices, BLOCK_SIZE=32 often wins due to superior memory coalescing within tiles. Auto-tuning validates this against BLOCK_SIZE=16.

This demonstrates that the theoretical maximum workgroup size is not always optimal.

The ratio of active warps (on GPUs) or wavefronts (on some NPUs) to the maximum number that can be resident simultaneously on a streaming multiprocessor (SM) or compute unit. It measures the utilization of parallel execution resources.

• High occupancy ensures the hardware can hide instruction and memory latency by switching between many active thread groups.
• Workgroup size directly impacts occupancy, as it determines how many thread groups can be scheduled based on shared resource limits (registers, shared memory).
• Optimal performance is often found at a balance point, not necessarily maximum occupancy, as overly small workgroups increase scheduling overhead, while overly large ones reduce the number of concurrent groups.

An optimization for memory access patterns where consecutive threads in a workgroup access consecutive memory addresses. This enables the memory controller to combine multiple accesses into a single, wide transaction.

• Coalescing is a hardware feature on NPUs/GPUs that dramatically increases effective memory bandwidth.
• Workgroup size and thread indexing must be designed to promote coalesced access. For example, if thread i accesses element A[i] in a contiguous array, accesses are coalesced.
• Poorly chosen workgroup sizes or memory access patterns can lead to uncoalesced accesses, where each thread triggers a separate memory transaction, severely bottlenecking performance.

A condition where threads within the same warp or wavefront (the smallest unit of parallel execution) follow different control flow paths due to conditional statements (e.g., if/else).

• Divergence causes serialization; the hardware must execute all divergent paths sequentially, disabling threads not on the current path.
• Workgroup organization influences divergence. If divergence correlates with thread ID within a warp, performance plummets. Algorithms should be designed for branch uniformity.
• Performance analysis tools measure branch efficiency to quantify the impact of divergence, which is a key consideration when tuning workgroup size and data partitioning.

The two primary performance regimes for a kernel, identifying the limiting factor for execution time.

• A kernel is compute-bound when its speed is limited by the arithmetic throughput of the NPU's cores. The ALUs are saturated, and increasing memory bandwidth yields little improvement.
• A kernel is memory-bound when its speed is limited by the memory subsystem's bandwidth or latency. The compute cores spend significant time stalled, waiting for data.
• Workgroup size tuning can shift the bottleneck. Larger workgroups may increase arithmetic intensity (ops/byte) moving towards compute-bound, while smaller groups may exacerbate memory pressure. Profiling tools use metrics like FLOPs/byte to diagnose the bound.

The multidimensional search domain defined by all tunable parameters for a computational kernel. For NPU/GPU kernels, this space includes:

• Workgroup size (X, Y, Z dimensions)
• Tile sizes for loop blocking
• Loop unrolling factors
• Vectorization width
• Use of shared/local memory

The size of the configuration space grows exponentially with the number of parameters, making exhaustive search infeasible. Auto-tuners like Kernel Tuner or TVM's AutoTVM use heuristic or model-based search (e.g., Bayesian Optimization) to efficiently navigate this space and find a performance-optimal configuration for a specific hardware target.

An analytical or machine-learned representation that predicts the execution time or resource usage of a kernel based on its parameters, input data, and hardware characteristics.

• Analytical models might estimate cycles based on known latencies, bandwidth limits, and operation counts.
• Empirical models are built by profiling many kernel configurations and learning a surrogate function (e.g., a Gaussian Process).
• These models are central to efficient auto-tuning. Instead of exhaustively measuring every configuration, the model predicts performance, guiding the search toward promising regions of the configuration space, including optimal workgroup sizes.