Memory Alignment

Memory alignment is a hardware-imposed constraint where a data object's starting memory address must be a multiple of its size (in bytes). For example, a 4-byte (32-bit) integer must be stored at an address divisible by 4 (e.g., 0x1000, 0x1004). This requirement exists because modern processors, especially RISC architectures and accelerators like NPUs, have memory buses and data paths optimized for aligned transfers. An unaligned access (e.g., a 4-byte integer starting at address 0x1001) may trigger a hardware exception (on some architectures) or force the processor to perform multiple, slower memory accesses to retrieve the data, which is known as a misaligned memory access penalty.

Proper alignment is critical for achieving single-cycle memory access. When data is aligned, the memory subsystem can fetch it in one operation over a wide bus. Misalignment causes significant performance degradation:

• Multiple Memory Transactions: The processor may need 2 or more reads/writes.
• Unaligned Load/Store Penalty: This can be 2-10x slower than an aligned access.
• Cache Inefficiency: It can waste cache space and bandwidth. For vectorized operations (SIMD) common in AI workloads, alignment is often mandatory. Instructions like Intel's SSE/AVX or ARM's NEON typically require data to be aligned to 16-byte or 32-byte boundaries for maximum throughput.

Compilers and programming languages provide mechanisms to enforce and query alignment.

• Alignment Specifiers: In C/C++, alignas(N) or __attribute__((aligned(N))) instructs the compiler to align a variable or struct member.
• Structure Padding: Compilers automatically insert padding bytes between struct members to ensure each member's natural alignment, which can increase memory usage.
• Allocation Functions: aligned_alloc(), posix_memalign(), and _mm_malloc() allocate memory with a specified alignment boundary.
• Offsetof: The offsetof macro can be used to inspect the byte offset of struct members, revealing compiler-added padding.

In Neural Processing Unit (NPU) and GPU programming, alignment is paramount for Direct Memory Access (DMA) engines and high-bandwidth memory (HBM) interfaces. Key considerations include:

• Tensor Layouts: Framework tensors (e.g., from PyTorch, TensorFlow) must often be aligned to specific boundaries (e.g., 128 bytes) for efficient transfer to the accelerator.
• Scratchpad Memory: Software-managed on-chip SRAM (scratchpad) often requires aligned accesses for parallel load/store units.
• Vendor SDKs: NPU SDKs (e.g., for NVIDIA, AMD, Intel, or custom ASICs) have strict alignment requirements for kernel arguments and buffer descriptors to enable zero-copy transfers and optimal memory coalescing.

Two major issues arise from misunderstanding alignment:

• Unaligned Pointers: Casting a char* buffer to an int* without ensuring alignment leads to undefined behavior or crashes.
• False Sharing: A severe multi-core performance bug where two unrelated variables (e.g., counters used by different threads) reside on the same cache line (typically 64 bytes). When one thread writes to its variable, it invalidates the entire cache line for other cores, causing excessive coherence traffic. The solution is to align and pad critical data structures to cache line boundaries using compiler directives.

Memory alignment directly influences data structure design for performance. Key strategies include:

• Ordering Struct Members: Place members in descending order of their alignment requirements (largest first) to minimize padding waste.
• Array of Structures (AoS) vs. Structure of Arrays (SoA): For SIMD processing, SoA—where each field is a separate, aligned array—often provides better spatial locality and alignment than AoS.
• Compiler Pragmas: Use #pragma pack with caution to reduce padding for network transmission, but be aware it may cause misaligned accesses and severe performance penalties on the receiving system.

A memory hierarchy organizes different types of memory (registers, caches, SRAM, DRAM) into a multi-level structure to balance access speed, capacity, and cost. Alignment requirements are often strictest at the fastest levels (e.g., registers, L1 cache) and propagate through the hierarchy. For NPUs, this includes global memory (HBM), shared memory (scratchpad), and register files.

A cache line is the smallest unit of data that can be transferred between the main memory and the cache. Its size (commonly 64 or 128 bytes) defines a fundamental alignment boundary. Unaligned accesses that straddle cache lines force two separate line fills, doubling memory traffic and latency. Efficient NPU programming requires structuring data to fit within and align to cache lines.

Padding is the insertion of unused bytes within a data structure to ensure proper alignment of its members. Compilers often add padding automatically. For example, in a C struct { char a; int b; }, 3 padding bytes may be inserted after a to align b on a 4-byte boundary. In performance-critical code, packed structures (removing padding) can save memory but may cause severe performance penalties due to misalignment.

Vector (SIMD) instructions process multiple data elements with a single instruction. These operations require the data address to be aligned to the vector width (e.g., 16-byte alignment for 128-bit SSE instructions). Misaligned vector loads may be emulated by the hardware with multiple micro-operations, causing significant slowdowns, or may raise a hardware exception. NPUs heavily utilize vector units, making alignment critical.

A strided access pattern refers to reading memory addresses separated by a constant interval (stride). Performance depends heavily on the relationship between the stride and cache line size or memory burst length. Unit stride (accessing consecutive elements) is optimal. Large, non-power-of-two strides can defeat hardware prefetchers and cause poor cache utilization, independent of alignment.

Modern memory controllers transfer data in fixed-size bursts (e.g., 64 bytes) over a wide bus. An aligned access that matches the burst size can be fulfilled in one transaction. A misaligned access may require two bursts, wasting bandwidth. This is a physical hardware constraint underlying software alignment rules. High Bandwidth Memory (HBM) on NPUs operates on similar principles.