TOPS (Tera Operations Per Second)

TOPS represents a theoretical maximum under ideal conditions, often using the simplest possible operation (e.g., INT8). Real-world inference throughput is always lower due to:

• Memory bandwidth bottlenecks: Data movement between memory and compute units.
• Model architecture inefficiencies: Non-optimal layer sizes for the hardware.
• Software stack overhead: Inefficiencies in drivers, kernels, and model compilers. A chip rated for 100 TOPS may deliver only 30-60 TOPS in a production workload.

TOPS values are meaningless without specifying the numerical precision of the operations. Performance scales inversely with precision:

• INT8 (8-bit integer): Highest TOPS, used for quantized inference.
• FP16 (16-bit floating point): Common for high-accuracy inference and training.
• FP32 (32-bit floating point): Lower TOPS, required for training and sensitive operations. A 100 TOPS (INT8) accelerator may only achieve 25 TOPS (FP16) and 6 TOPS (FP32) on the same silicon.

The underlying hardware architecture defines how TOPS are achieved. Key components include:

• Multiply-Accumulate (MAC) Units: The physical circuits that perform the core neural network operations. TOPS = (Number of MACs) × (Clock Frequency) × 2 (for fused multiply-add).
• Memory Hierarchy: On-chip SRAM (scratchpad) size and bandwidth determine how efficiently data feeds the MACs.
• Dataflow Engines: Architectures like weight-stationary or output-stationary optimize data reuse to minimize off-chip memory access, impacting achievable utilization.

For edge AI, the performance-per-watt metric (TOPS per Watt) is often more critical than raw TOPS. It is determined by:

• Process Node: Smaller semiconductor nodes (e.g., 5nm vs. 28nm) offer more operations per joule.
• Circuit Design: Techniques like clock gating and power gating deactivate unused circuits.
• Thermal Design Power (TDP): The chip's power envelope limits sustainable performance. A 30 TOPS/W accelerator in a 5W envelope delivers 150 TOPS maximum.

A model compiler (e.g., TVM, Apache TVM, proprietary SDKs) is essential to approach the theoretical TOPS. It performs:

• Graph Optimization: Fusing layers, eliminating redundant operations.
• Kernel Scheduling: Mapping computational graphs to the hardware's execution units.
• Memory Planning: Minimizing data movement through smart tensor tiling and caching. Poor compilation can reduce effective throughput by an order of magnitude versus the peak TOPS rating.

TOPS should be compared cautiously across different chip types:

• NPU vs. GPU: An NPU's TOPS are for dedicated neural network primitives; a GPU's TOPS may include general matrix math but with higher power draw.
• Peak vs. Sustained: NPUs often sustain a higher percentage of their peak TOPS than GPUs on typical AI workloads.
• System-Level Performance: The System-on-Chip (SoC) integration (CPU, NPU, ISP, memory) determines the final application latency, not the NPU's TOPS in isolation.

FLOPS measures the rate of floating-point calculations a processor can perform. It is the foundational unit for TOPS, which scales it to trillions (tera) of operations. While TOPS is often cited for AI accelerators, FLOPS is critical for workloads requiring high numerical precision, such as scientific computing and model training.

• Single-Precision (FP32) vs. Half-Precision (FP16): Performance varies drastically by data type.
• Theoretical vs. Achievable: Like TOPS, peak FLOPS is a theoretical maximum rarely sustained in practice due to memory bottlenecks and architectural inefficiencies.

The power envelope is the total electrical power budget allocated for a device or system. For edge AI hardware, this is a primary constraint that directly limits achievable TOPS.

• Performance-Per-Watt: The key efficiency metric, often expressed as TOPS per watt (TOPS/W).
• Thermal Design Power (TDP): Closely related; defines the maximum heat the cooling system must dissipate.
• Dynamic Trade-off: Processors use techniques like Dynamic Voltage and Frequency Scaling (DVFS) to stay within the power envelope, often throttling performance (and effective TOPS) to manage heat and battery life.

Memory bandwidth is the rate at which data can be read from or written to memory, measured in gigabytes per second (GB/s). It is often the critical bottleneck limiting real-world TOPS performance.

• The "Memory Wall": High-TOPS accelerators can stall waiting for model weights and activations to be fetched from memory.
• On-Chip SRAM vs. Off-Chip DRAM: Fast, small SRAM caches are essential to feed compute units and mitigate bandwidth limitations.
• Architectural Impact: Designs like Compute-in-Memory (CIM) aim to eliminate this bottleneck by performing calculations within the memory array itself.

The numerical precision used for calculations dramatically impacts quoted TOPS figures. Accelerators often have separate performance ratings for different data types.

• INT8 (8-bit Integer): Used for quantized inference, offering the highest TOPS ratings due to simpler, faster arithmetic. Common for computer vision and NLP tasks.
• FP16 (16-bit Floating-Point): Used for higher-precision workloads, like some generative AI or scientific inference. Delivers lower TOPS than INT8 but greater numerical fidelity.
• Architectural Support: Modern NPUs and GPUs have separate hardware lanes optimized for each precision (e.g., Tensor Cores for FP16/INT8).

Heterogeneous computing is a system architecture that uses a mix of specialized processors—such as CPUs, GPUs, NPUs, and DSPs—to execute different parts of a workload where they are most efficient. A high-TOPS NPU is one component within this ecosystem.

• Workload Orchestration: The system software must efficiently partition tasks (e.g., pre-processing on CPU, inference on NPU).
• Amdahl's Law: The overall system speedup is limited by the fraction of the workload that cannot be parallelized on the accelerator.
• Unified Memory: Architectures like UMA (Unified Memory Access) reduce data copying overhead between different processing units.