Scratchpad Memory

The defining characteristic of scratchpad memory is its software-managed nature. Unlike a traditional hardware cache, which uses automatic algorithms (e.g., LRU) to decide what data to keep, the programmer or compiler must explicitly control all data movement into and out of the scratchpad. This shifts the complexity of optimizing data locality from hardware logic to software, providing deterministic performance at the cost of increased programming effort. The compiler must analyze data access patterns and insert explicit DMA (Direct Memory Access) transfers or load/store instructions to populate the scratchpad.

Because scratchpad memory is on-chip SRAM and access is controlled explicitly by software, it offers predictable, low-latency access. There is no concept of a cache miss; if data is in the scratchpad, the access time is constant. This determinism is critical for real-time and latency-sensitive applications common in NPU workloads, such as processing individual layers of a neural network within a strict inference deadline. The absence of tag-checking and replacement logic found in caches further simplifies the memory controller, reducing access energy and cycle time.

Effective use of scratchpad memory relies on the algorithmic technique of tiling (or blocking). Large tensors or matrices are broken into smaller tiles that fit within the scratchpad's limited capacity. A computational kernel loads a tile into the scratchpad, performs all necessary operations on it, writes results back to main memory (e.g., HBM), and then loads the next tile. This pattern maximizes data reuse within the fast on-chip memory, minimizing costly accesses to slower off-chip DRAM. The compiler's ability to automate this tiling and data movement is a key optimization in frameworks like MLIR and TVM.

In a typical NPU memory hierarchy, scratchpad memory sits between processor registers and large, off-chip High Bandwidth Memory (HBM). It is often organized as Shared Memory or Local Data Memory that is accessible by a group of processing cores (e.g., a tensor core array). This placement makes it ideal for storing:

• Activation maps between neural network layers
• Weight tiles for convolutional or linear layers
• Intermediate results from partial computations Its size is a critical hardware design trade-off, typically ranging from hundreds of kilobytes to a few megabytes per core cluster.

Scratchpad memory and hardware caches represent two different solutions to the memory wall problem.

Scratchpad (SPM):

• Pro: Deterministic latency, no miss penalty, higher energy efficiency, simpler hardware.
• Con: Requires complex compiler analysis and explicit data management.

Hardware Cache:

• Pro: Transparent to software, adapts to dynamic access patterns.
• Con: Non-deterministic performance due to misses, complex tag RAM and logic, coherence overhead in multi-core systems.

Many modern NPUs and GPUs use a hybrid approach, featuring both small L1 caches for instruction/ scalar data and larger software-managed scratchpads (e.g., Shared Memory in CUDA) for bulk tensor data.

Leveraging scratchpad memory is a primary task for NPU compilers (e.g., for Google's TPU, NVIDIA's TensorRT, or Intel's OpenVINO). The compiler's job includes:

• Memory Allocation: Statically assigning regions of the scratchpad to specific tensors or tiles.
• Scheduling DMA Transfers: Overlapping data movement with computation to hide latency.
• Coalescing Accesses: Ensuring memory transactions are efficient for the underlying DRAM and interconnect (e.g., CXL).
• Kernel Fusion: Merging operations to keep intermediate results in the scratchpad, avoiding round trips to main memory. This compiler-driven optimization is a cornerstone of achieving peak FLOPS/Watt on AI accelerators.

Cache memory is a small, high-speed hardware-managed memory that transparently stores copies of frequently accessed data from main memory. Unlike a scratchpad, its management (what data is stored and when it is evicted) is handled automatically by hardware logic based on access patterns.

• Key Difference: Caches use hardware-managed policies (e.g., LRU) for data placement and replacement, while scratchpads rely on explicit software directives.
• Trade-off: Caches simplify programming but introduce unpredictability (cache misses) and hardware overhead (tag storage, coherence logic). Scratchpads offer deterministic, low-latency access at the cost of requiring explicit data movement in software.

High Bandwidth Memory (HBM) is a 3D-stacked DRAM technology that provides extremely high data transfer rates in a compact physical footprint. It typically serves as the high-capacity, off-chip global memory for accelerators like NPUs and GPUs.

• Relationship to Scratchpad: The scratchpad (SRAM) is the small, on-chip, ultra-fast memory. HBM is the large, off-chip, high-bandwidth pool. Efficient NPU programming involves strategically staging data from HBM into the scratchpad for computation, minimizing costly off-chip accesses.
• Performance Impact: The bandwidth of the HBM interface (often hundreds of GB/s to over 1 TB/s) is a critical bottleneck. Scratchpad usage directly reduces pressure on this interface.

Direct Memory Access (DMA) is a hardware mechanism that allows peripherals or accelerators to transfer data directly between memory regions (e.g., from system RAM to accelerator memory) without continuous CPU intervention.

• Critical for Scratchpads: DMA engines are essential for efficiently prefetching data from slow, high-latency global memory (like HBM) into the software-managed scratchpad before a compute kernel executes. This allows computation and data transfer to overlap.
• Use Case: A compiler or programmer schedules a DMA transfer to load the next tile of a matrix into the scratchpad while the NPU cores are still computing on the current tile.

Memory latency is the time delay between a request for data and its availability. It is a primary performance limiter, known as the "memory wall."

• Scratchpad's Role: Scratchpad memory exists specifically to mitigate latency. Access times are typically on the order of tens of cycles, compared to hundreds of cycles for off-chip HBM or DDR memory.
• Determinism: Because software explicitly controls scratchpad contents, latency is predictable and consistent, unlike caches where a cache miss introduces highly variable and severe penalties.

Spatial locality (using nearby data) and temporal locality (re-using the same data) are fundamental program behavior principles that guide memory hierarchy design.

• Scratchpad Exploitation: Effective use of a scratchpad requires the programmer or compiler to explicitly identify and exploit these localities. Data with high temporal locality (e.g., weights reused across many operations) is a prime candidate for scratchpad residency.
• Contrast with Cache: Caches implicitly try to exploit locality based on past access patterns. Scratchpads require explicit algorithmic design (e.g., tiling/blocking) to structure computations to maximize locality within the limited on-chip memory.

Unified memory (e.g., CUDA Unified Memory, AMD's ROCm HIP) is a programming model that presents a single, coherent address space shared by the CPU and an accelerator (GPU/NPU). The system hardware and software automatically manage data movement.

• Philosophical Opposite: Unified memory aims for programming simplicity by hiding the memory hierarchy. Scratchpad programming embraces complexity for control, requiring the programmer to explicitly manage data placement.
• Performance Consideration: While unified memory simplifies code, it often relies on page fault-driven migration, which can introduce latency spikes. For peak performance on NPUs, explicit scratchpad management with DMA is usually required.