Auto-Tuning

Tile sizes define the dimensions of data blocks processed within faster memory hierarchies like shared memory or registers. The auto-tuner searches for tile dimensions (e.g., 32x32, 64x64, 128x128) that maximize data reuse and fit within hardware constraints.

• Goal: Balance data locality against register pressure and shared memory capacity.
• Impact: Poor tiling leads to excessive global memory traffic, making the kernel memory-bound. Optimal tiling can improve performance by 10x or more.
• Example: For a matrix multiplication, the tuner experiments with block sizes for the M, N, and K dimensions to minimize DRAM accesses.

The loop unroll factor specifies how many iterations of a loop body are duplicated to reduce loop control overhead and increase instruction-level parallelism (ILP).

• Mechanism: Replicates loop body statements, decreasing branch instructions and exposing more independent operations for the scheduler.
• Trade-off: Higher unrolling increases register pressure and can lead to register spilling. The auto-tuner finds the factor that maximizes ILP without causing spills.
• Hardware Consideration: Optimal factors are often tied to the processor's issue width and register file size.

Vector width determines the number of data elements processed by a single SIMD (Single Instruction, Multiple Data) or SIMT instruction. Auto-tuners configure this for kernel vectorization.

• Objective: Fully utilize the hardware's vector units (e.g., 4-wide, 8-wide, 16-wide).
• Constraint: Limited by data type (FP32, FP16, INT8) and the underlying instruction set architecture.
• Process: The tuner generates code with different vectorization pragmas or intrinsic mappings and measures throughput.

These parameters define the parallelism granularity on accelerators like GPUs/NPUs. Thread block size (e.g., 128, 256, 512 threads) and grid layout determine how work is mapped to hardware execution units.

• Kernel Occupancy: Block size directly impacts occupancy, the number of active warps per streaming multiprocessor.
• Resource Limits: Block size is constrained by shared memory per block and register count per thread.
• Auto-Tuner Role: Exhaustively searches the legal space of block/grid dimensions to maximize occupancy and hide memory latency.

For software pipelining, auto-tuners adjust the initiation interval and the schedule of operations across loop iterations to hide instruction and memory latency.

• Initiation Interval (II): The number of cycles between starting successive loop iterations. The tuner aims for the minimum possible II.
• Schedule Search: The tuner explores different orderings of load, compute, and store operations to create an efficient pipeline that minimizes stalls.
• Application: Critical for deeply pipelined architectures and loops with long latency operations.

Auto-tuners optimize parameters governing memory coalescing and prefetching behavior to saturate memory bandwidth.

• Coalescing Configuration: Adjusts data layout or access strides to ensure threads within a warp access contiguous memory addresses.
• Prefetch Distance: Determines how many iterations ahead data is fetched into registers or caches. The tuner finds the distance that hides DRAM latency without polluting caches.
• Shared Memory Bank Conflict Avoidance: Configures padding or access patterns to avoid serialized accesses to shared memory banks.