Compute-in-Memory

Memristors (memory resistors) are non-volatile devices whose resistance can be programmed. Arranged in dense crossbar arrays, they can perform analog matrix-vector multiplication in a single step by exploiting Ohm's law (current) and Kirchhoff's law (summation). This is the physical embodiment of in-memory computing, ideal for accelerating the dense linear algebra of neural networks with extreme energy efficiency.

• Key Advantage: High density and non-volatility enable storage of weights directly at the compute site.
• Challenge: Device variability and limited precision (typically < 8-bit) require robust training algorithms.
• Example: Systems from Mythic AI and Weebit Nano use this approach.

This approach uses standard Static Random-Access Memory bitcells, the same technology in CPU caches, but modifies the peripheral circuitry to enable bit-wise logical operations directly on stored data. It performs computation in the digital domain, offering higher precision and compatibility with existing CMOS logic than analog methods.

• Key Advantage: High speed, deterministic digital operation, and compatibility with standard chip fabrication processes.
• Common Operation: Used for binary/ternary neural networks or as a foundational block for more complex digital arithmetic.
• Example: Research from IBM and TSMC demonstrates SRAM-CIM macros for AI inference.

Resistive Random-Access Memory and Phase-Change Memory are two leading non-volatile memory technologies used for CIM. Both work by changing the physical state of a material to alter its resistance.

• ReRAM: Changes resistance via formation/rupture of conductive filaments. Offers good scalability and speed.
• PCM: Changes resistance by switching between amorphous (high-resistance) and crystalline (low-resistance) phases. Offers high endurance.
• Application: Like memristors, they are deployed in crossbars for analog computation. Companies like Intel (with PCM-based Loihi neuromorphic chips) and Panasonic have pioneered this path.

An adjacent concept where simple processing units are integrated into or near Dynamic RAM banks. While not performing computation directly in the memory array like analog CIM, it drastically reduces data movement by bringing compute to the data. This is sometimes called near-memory computing.

• Mechanism: Adds lightweight arithmetic logic units (ALUs) within the DRAM chip or on a logic layer in 3D-stacked memory (e.g., HBM).
• Use Case: Excellent for bandwidth-bound operations like vector additions and reductions.
• Example: Samsung's HBM-PIM and SK hynix's AiM are commercial products implementing this technology.

An emerging technology that integrates a ferroelectric material into the gate stack of a standard transistor. The polarization state of the ferroelectric layer acts as a non-volatile weight, and the transistor itself can perform the multiply-accumulate operation.

• Key Advantage: Combines logic and memory in a single, scalable device built on existing transistor technology, promising high density and energy efficiency.
• Status: Primarily in advanced research stages at institutions like GlobalFoundries and NaMLab.
• Potential: Seen as a promising path for deeply scaled, logic-compatible CIM.

Uses light instead of electricity to perform computations. Weights can be encoded in the phase or amplitude of light passing through programmable optical components (e.g., Mach-Zehnder interferometers, phase-change materials). Matrix multiplication is performed via the physics of light interference as it passes through these tuned elements.

• Key Advantage: Ultra-high bandwidth and speed with minimal heat generation, as photons do not interact like electrons.
• Challenge: Complex packaging, conversion between electronic and photonic domains, and precision control.
• Example: Startups like Lightmatter and Lightelligence are developing photonic AI accelerators.

CiM is a foundational technology for deploying neural networks on battery-powered edge devices and microcontrollers. By eliminating the energy cost of moving weights from DRAM, CiM enables always-on sensing and real-time inference for applications like keyword spotting, anomaly detection in industrial sensors, and wake-word detection in smart speakers. This makes advanced AI feasible in scenarios with severe power budgets (<1mW).

CiM arrays are uniquely suited to accelerate the dense linear algebra that forms the computational core of many neural network layers.

• Fully-Connected (Dense) Layers: The matrix-vector multiplication (y = Wx) maps directly onto a crossbar array, where weights W are stored as conductance values and the input vector x is applied as voltages.
• Convolutional Layers: Can be unrolled and mapped to large matrix multiplications, which CiM can process in a highly parallel manner.
• Attention Mechanisms: The key query-dot product operations in transformer attention heads are another form of vector similarity search well-suited to analog CiM.

Beyond neural networks, CiM can accelerate fundamental database and search primitives by performing computation directly on stored data.

• Similarity Search: Analog CiM crossbars can compute the dot-product between a query vector and all stored database vectors in parallel, accelerating nearest neighbor search for recommendation systems or vector database lookups.
• Graph Analytics: Operations like breadth-first search or PageRank, which involve traversing adjacency matrices stored in memory, can see significant speedups by reducing data movement.
• Data Filtering & Selection: Parallel bitwise operations within memory arrays can rapidly filter records based on multiple criteria.

CiM architectures show promise for accelerating specific classes of scientific and numerical workloads.

• Sparse Matrix-Vector Multiplication (SpMV): A bottleneck in many simulations (e.g., computational fluid dynamics, finite element analysis). CiM architectures with sparsity encoding support can skip zero-valued computations, offering energy-efficient acceleration.
• Iterative Solvers: Algorithms like Conjugate Gradient, which rely heavily on SpMV, can benefit from the reduced data movement latency of CiM.
• Molecular Dynamics & Quantum Chemistry: These fields involve calculating potentials and forces, which often reduce to n-body problems with structured linear algebra kernels.

The parallel compute fabric of CiM is ideal for low-level signal processing tasks that are precursors to higher-level AI analysis.

• Digital Signal Processing (DSP): Filters (FIR, IIR), Fast Fourier Transforms (FFT), and correlation operations involve repetitive multiply-accumulate patterns that map efficiently to CiM arrays.
• Image Processing Kernels: Operations like convolution for edge detection, blurring, or sharpening can be executed in-memory on the raw pixel data, reducing latency before feeding into a vision model.
• Baseband Processing in Wireless: Tasks like channel estimation and equalization in 5G/6G radios involve linear algebra that could be offloaded to dedicated CiM units for lower power consumption.

Emerging research explores CiM for accelerating cryptographic operations and enhancing hardware security.

• Physical Unclonable Functions (PUFs): The inherent analog variations in memristor-based CiM crossbars can be used to generate device-unique digital fingerprints for hardware authentication.
• Homomorphic Encryption (HE): Some HE schemes involve polynomial arithmetic over large matrices, a potentially CiM-acceleratable workload, though precision requirements are a significant challenge.
• Random Number Generation: The stochastic switching behavior of certain resistive memory devices can be harnessed to generate true random numbers within the memory array itself.

The Von Neumann bottleneck is the fundamental performance limitation in traditional computer architectures where the central processing unit (CPU) and memory are separate. Data must be constantly shuttled between them over a bus, which consumes significant energy and time. Compute-in-Memory directly addresses this bottleneck by performing calculations within the memory array itself, eliminating this costly data movement for key operations like matrix-vector multiplication.

The memory wall describes the growing performance gap between processor speed and memory access speed. While compute units have scaled rapidly, memory bandwidth and latency improvements have lagged. This imbalance makes data access the primary bottleneck for many workloads, especially data-intensive neural network inference. CIM architectures attack the memory wall by collapsing the physical and logical separation between where data is stored and where it is processed.

A memristor (memory resistor) is a fundamental, non-volatile circuit element whose electrical resistance depends on the history of current that has flowed through it. This property allows it to store analog values (e.g., neural network weights) and perform in-memory analog computation using Ohm's law and Kirchhoff's law. Memristor crossbar arrays are a leading candidate for building dense, energy-efficient CIM chips for deep learning.

Processing-in-Memory is a broader architectural category that includes Compute-in-Memory. While CIM typically refers to performing computation using the memory cells themselves (e.g., analog computation in a memristor array), PIM can also involve placing simple digital processing units (ALU, logic) inside or near the memory bank. The core principle of both is to reduce data movement by bringing computation closer to the data.

Analog AI refers to performing neural network computations using analog electronic signals (current, voltage, charge) rather than digital binary values. This approach, often implemented with memristor-based CIM, leverages the native physics of the device for multiplication and accumulation. It offers extreme energy efficiency but introduces challenges with noise, precision, and device variability that require sophisticated circuit and algorithm co-design.

SRAM-based Compute-in-Memory uses standard digital Static Random-Access Memory bitcells as the foundation for in-memory computation. By modifying the peripheral circuitry and sense amplifiers, these arrays can perform bitwise logical operations or parallel search operations within the memory block. This digital approach offers high precision and compatibility with existing CMOS processes but is generally less dense and efficient for large-scale matrix multiplication than analog approaches.