Tensor Core

The core operation of a Tensor Core is the matrix multiply-accumulate (MMA). It computes D = A * B + C, where A, B, C, and D are matrices. Crucially, Tensor Cores perform this operation in mixed precision: they typically take inputs A and B in a lower precision format (like FP16, BF16, INT8, or INT4) and accumulate the products into a higher precision accumulator (like FP32 or INT32). This reduces memory bandwidth and increases computational density while maintaining numerical fidelity for training and inference.

• Example: NVIDIA's Ampere architecture Tensor Cores can perform an MMA operation on a 16x16x16 matrix tile per clock cycle using FP16 inputs and FP32 accumulation.

Tensor Cores are not simply faster CUDA cores; they are a separate, dedicated execution unit on the chip. CUDA cores are designed for general-purpose, single-precision floating-point (FP32) and integer operations with high flexibility. Tensor Cores are hardwired, systolic arrays optimized for the specific dataflow patterns of matrix multiplication. This specialization allows them to achieve orders of magnitude higher throughput for GEMM (General Matrix Multiply) operations, which are foundational to deep learning (convolution, fully-connected layers, attention). The two units work in concert, with CUDA cores handling control flow, element-wise ops, and data preparation.

Tensor Cores are implemented using a systolic array architecture. This is a grid of tightly coupled processing elements (PEs) where data flows rhythmically between adjacent PEs, minimizing data movement and maximizing reuse. In a systolic array for MMA:

• Matrix A's rows are pumped horizontally across the array.
• Matrix B's columns are pumped vertically.
• Each PE holds a partial sum, which accumulates as the operands flow through. This design is exceptionally efficient for dense linear algebra because it keeps data on-chip, reducing energy-intensive accesses to caches and DRAM. It is the hardware embodiment of the kernel fusion principle at the arithmetic unit level.

Modern Tensor Cores support multiple precision modes to match algorithmic requirements:

• Training: TF32, BF16, FP16 with FP32 accumulation.
• Inference: INT8, INT4, and even binary (1-bit) for extreme efficiency.

A key advanced feature is structured sparsity. Starting with Ampere architecture, Tensor Cores can exploit fine-grained (2:4) sparsity patterns in weight matrices. During inference, the hardware can skip computations on zero values, effectively doubling the throughput for sparse models. This requires model weights to be pruned to the specific 2:4 pattern (2 non-zero values per block of 4).

The inclusion of Tensor Cores dramatically shifts the roofline model for matrix-heavy workloads. They raise the peak arithmetic intensity (operations per byte) attainable by performing more computations on data once it's in the chip. A workload that was memory-bound on CUDA cores (limited by DRAM bandwidth) can become compute-bound on Tensor Cores, saturating the new, much higher compute ceiling. For example, the theoretical peak FP16 Tensor Core throughput on an NVIDIA H100 is over 1,000 TFLOPS, compared to ~60 TFLOPS for FP32 on its CUDA cores. Effective use requires careful memory coalescing and data layout to keep the systolic array fed, making compiler optimizations like loop tiling and kernel fusion critical.

To achieve peak Tensor Core performance, software must satisfy specific hardware constraints:

• Data Precision: Operations must use supported types: FP16, BF16, TF32, INT8, or FP8.
• Matrix Dimensions: For FP16/BF16/TF32, matrix sizes (M, N, K) should be multiples of 8. For INT8, multiples of 16.
• Data Alignment: Pointers to matrix data in global memory must be aligned to 16-byte boundaries.
• Kernel Occupancy: Sufficient concurrent warps must be scheduled to hide Tensor Core pipeline latency. This is managed by the compiler and runtime via register allocation and shared memory usage. Failure to meet these conditions causes a fallback to slower CUDA Cores.

The Streaming Multiprocessor is the fundamental programmable computing core in NVIDIA GPU architectures. It contains:

• CUDA Cores for general-purpose FP32/INT32 operations
• Tensor Cores for specialized matrix math
• Shared Memory/L1 Cache for fast data exchange between threads
• Warp Schedulers for instruction dispatch Tensor Cores are embedded within SMs, and efficient kernel execution requires coordinating their use with the SM's other resources to avoid bottlenecks.

Matrix Multiply-Accumulate is the fundamental mathematical operation performed by a Tensor Core. A single MMA instruction computes: D = A * B + C Where A and B are input matrices, C is an accumulator matrix, and D is the result. Key characteristics:

• Mixed-Precision: Typically uses FP16/BF16 inputs with FP32 accumulation for numerical stability.
• High Throughput: Processes small, fixed-size matrix tiles (e.g., 4x4, 8x4, 16x8) per clock cycle.
• Fused Operations: The multiply and accumulate are a single, indivisible hardware operation, crucial for deep learning training and inference.

Tensor Core instructions are warp-level primitives, meaning they are issued and executed collectively by all 32 threads in a CUDA warp. This differs from scalar operations where threads work independently. Implications:

• Cooperative Execution: Threads in a warp must cooperate to load the input matrix fragments (A, B) and store the result (D).
• Data Distribution: The matrix tiles are distributed across the warp's threads, following specific layout patterns (e.g., row-major, column-major).
• Synchronization: No explicit synchronization is needed within the warp for the MMA operation itself, as it's hardware-coordinated.

The Tensor Memory Accelerator is a dedicated hardware unit introduced in NVIDIA's Hopper architecture that works in concert with Tensor Cores. Its primary function is to asynchronously transfer large, multi-dimensional tensor tiles between global memory and shared memory. Benefits:

• Overlaps Compute & Data Movement: TMA can load the next tile of data while Tensor Cores process the current tile.
• Efficient 2D/3D Patterns: Optimized for strided, nested data accesses common in convolutional and transformer layers.
• Reduces SM Overhead: Offloads complex address calculation and memory transaction coalescing from the SM's cores.

Modern Tensor Cores (e.g., Ampere architecture and later) include hardware support for structured sparsity. This feature exploits the fact that many weights in a pruned neural network are zero. The mechanism:

• A 2:4 sparsity pattern is enforced: for every block of 4 values, at least 2 must be zero.
• The Tensor Core skips computation on the zero values, effectively doubling the throughput for matrix operations on sparse matrices.
• Requires a pruning & fine-tuning workflow to create compatible models and uses specialized sparse tensor formats in memory.