Memory Bandwidth Bottleneck

The fundamental architectural limitation where a shared bus between the CPU and memory creates a data transfer choke point. Compute throughput is throttled by the rate at which operands can be fetched and results stored, not by the processor's theoretical peak speed. This is especially acute in deep learning, where weight matrices must be streamed continuously from DRAM to the processing elements.

A visual performance model that plots attainable operations per second against arithmetic intensity (FLOPs per byte). The sloped region represents the memory-bound regime where performance is dictated by bandwidth, not peak compute. For workloads like element-wise activation functions or depthwise convolutions, the arithmetic intensity is low, pinning the application firmly against the bandwidth ceiling.

The primary mitigation strategy involves maximizing temporal and spatial locality. Techniques include:

• Tiling: Partitioning tensors into blocks that fit into on-chip SRAM caches.
• Operator Fusion: Combining consecutive layers (e.g., convolution + batch norm + ReLU) to avoid round-trips to DRAM.
• Kernel Design: Writing custom GPU kernels that stage data through shared memory explicitly.

High-Bandwidth Memory (HBM) stacks DRAM dies vertically using through-silicon vias (TSVs) and places them on a silicon interposer next to the compute die. This provides a massively wide bus (1024-bit or greater) compared to traditional GDDR interfaces. HBM2e and HBM3 are standard in datacenter GPUs and AI accelerators, trading higher cost and manufacturing complexity for a critical reduction in the bottleneck.

On embedded platforms like FPGAs and microcontrollers, the bottleneck is severe due to limited external DRAM interfaces and low-power LPDDR memory. Model compression techniques—quantization, pruning, and knowledge distillation—are not just about reducing storage; they are essential to shrink the working set to fit into on-chip SRAM, effectively bypassing the external memory bottleneck entirely.

The critical ratio of computation to communication that a specific hardware platform requires to transition from memory-bound to compute-bound. For a modern GPU with 2 TB/s bandwidth and 1000 TFLOPS, the threshold is roughly 500 FLOPs per byte. Any operation below this ratio wastes compute cycles waiting for data. Understanding this threshold is essential for selecting the right accelerator for a given neural network architecture.

A hardware mechanism that allows peripherals to transfer data directly to and from system memory without continuous CPU intervention. In RF fingerprinting pipelines, DMA controllers move digitized IQ samples from ADCs into processor memory while the CPU simultaneously executes inference.

• Eliminates processor stalls during large data transfers
• Critical for streaming high-bandwidth RF data at gigasample rates
• Enables zero-copy architectures when combined with shared memory regions
• Common in FPGA-based SDR platforms using AXI DMA IP cores

A graph optimization technique that combines multiple discrete neural network layers into a single compute kernel. By fusing convolution, batch normalization, and activation functions, intermediate tensor reads and writes to external memory are eliminated.

• Reduces memory bandwidth consumption by up to 40% in CNN pipelines
• Implemented natively in TensorRT, OpenVINO, and ONNX Runtime
• Particularly effective for 1D convolutional layers used in IQ waveform processing
• Trade-off: increased register pressure and kernel complexity

A model compression strategy that removes redundant or near-zero parameters from a neural network. Unstructured pruning creates sparse weight matrices that reduce the total data volume transferred from memory during inference.

• Can achieve 50-90% sparsity with minimal accuracy loss
• Requires sparse matrix multiplication hardware support for actual speedup
• Structured pruning removes entire channels or filters for direct memory footprint reduction
• Often combined with quantization for compounding bandwidth savings

A technique that uses different numerical precisions (FP16, INT8, FP32) for different layers within a single model. Sensitive layers retain higher precision while computationally intensive layers use reduced bit widths.

• Halves memory bandwidth requirements when moving from FP32 to FP16
• INT8 quantization reduces bandwidth by 4x compared to FP32
• Critical for deploying large transformer-based signal classifiers on edge hardware
• Supported by NVIDIA Tensor Cores and AMD XDNA accelerators

A data management paradigm where pointers to memory buffers are passed between processing stages rather than physically copying data. In RF pipelines, ADC samples are written once to a shared memory region accessible by both the DSP chain and the AI inference engine.

• Eliminates redundant memcpy operations that saturate memory buses
• Requires hardware support for unified memory architectures
• Implemented via AXI4-Stream interfaces in FPGA designs
• Essential for real-time wideband spectrum monitoring applications

A distributed inference strategy where different model layers are assigned to sequential hardware stages. Each stage processes a different input sample simultaneously, keeping all compute units busy while amortizing memory access patterns.

• Converts memory-bound operations into compute-bound operations
• Effective when a single accelerator cannot hold the entire model in local SRAM
• Common in FPGA deployments where each layer is a separate hardware block
• Requires careful balancing of stage execution times to avoid bubbles