FLOPs (Floating Point Operations)

A FLOP is a single floating-point arithmetic operation, such as an addition, subtraction, multiplication, or division. In deep learning, the vast majority of FLOPs come from matrix multiplications and convolutions. For example, multiplying two n x n matrices requires approximately 2n³ FLOPs. This count excludes memory access operations, focusing purely on the computational kernel.

FLOP counts are typically reported for a single forward pass of one data sample through the model. The backward pass during training, which computes gradients via backpropagation, generally requires roughly twice the FLOPs of the forward pass. Therefore, a model with 100 GFLOPs for inference may require ~300 GFLOPs per sample for a full training step (forward + backward).

FLOPs measure theoretical peak computation, not actual runtime. Real-world speed depends on:

• Hardware FLOP/s (FLOPS): The processor's maximum operations per second (e.g., an NVIDIA H100 delivers ~67 TFLOPS for FP16).
• Memory Bandwidth: If operations are memory-bound, high FLOP counts don't translate to speed.
• Kernel Optimization: Efficiently mapped operations (e.g., using Tensor Cores) achieve a higher percentage of peak FLOPS. Thus, a lower-FLOP model may run slower if it's poorly optimized for the hardware.

FLOPs serve as a first-order proxy for energy cost and carbon footprint. Each floating-point operation consumes a measurable amount of energy on silicon. Training large models like GPT-3 (estimated at 3.14e23 FLOPs) directly correlates with massive megawatt-hour consumption. Optimizing FLOPs is a primary lever for reducing the environmental impact and cloud compute bills of AI projects.

FLOPs have significant limitations:

• Ignores Activation Memory: Does not account for the cost of storing intermediate activations, which can be the true bottleneck.
• No Parallelism Insight: A 100 GFLOP model with high parallelism may run faster than a 50 GFLOP sequential model.
• Operation Mix: Treats all FLOPs equally, though some hardware (e.g., GPUs) execute multiplications and additions at different speeds. It is best used alongside metrics like parameter count, activation size, and measured latency.

FLOPs are heavily influenced by architectural choices:

• Dense Layers: FLOPs scale with the product of input and output neurons.
• Convolutional Layers: FLOPs = (H * W * C_in * K_h * K_w * C_out) for output size HxW, kernel size K, and channels C.
• Attention Layers: The self-attention mechanism in Transformers scales with O(n²d) for sequence length n and dimension d, making it a major FLOP contributor in LLMs. Architects trade these components to balance FLOPs, accuracy, and model capacity.

FLOPs are the primary metric for sizing inference hardware. By calculating a model's FLOPs per forward pass, engineers can match it to hardware with sufficient FLOPS (Floating Point Operations Per Second) capacity to meet target latency. For example, a 100 GFLOP model requires hardware capable of at least 100 GFLOPS to run in one second. This analysis prevents costly over-provisioning and identifies bottlenecks, guiding decisions between GPUs, TPUs, and NPUs.

FLOPs serve as a key constraint in automated Neural Architecture Search. The search algorithm explores a vast space of potential model designs (e.g., different layer depths, widths, kernel sizes) and uses FLOPs as a proxy for computational efficiency. This allows for the discovery of Pareto-optimal architectures that balance high accuracy with low computational cost, crucial for deploying models on edge devices or under strict inference budget constraints.

FLOPs enable precise forecasting of cloud compute expenses. The total FLOPs for a training run is calculated as: Total Training FLOPs = (FLOPs per forward/backward pass) * (Number of training samples) * (Number of epochs) This figure, combined with the FLOPS/$ rate of cloud instances, provides a direct cost estimate. For inference, FLOPs per query multiplied by expected query volume predicts sustained infrastructure costs, forming the basis for Service Level Objective (SLO) planning and Total Cost of Ownership (TCO) models.

FLOPs provide a standardized, hardware-agnostic basis for comparing the efficiency of different model architectures. When evaluating models with similar accuracy, the one with lower FLOPs is inherently more efficient. This is critical for leaderboards and research, moving beyond accuracy-only metrics to consider the computational cost of performance. For instance, comparing a Vision Transformer (ViT) to a ConvNeXt model requires analyzing both Top-1 accuracy and GFLOPs to understand the performance/cost trade-off.

FLOPs analysis identifies the most computationally expensive layers (e.g., large fully-connected or attention layers), guiding targeted application of compression methods:

• Pruning: Removing low-weight neurons/channels from high-FLOP layers yields the greatest reduction.
• Quantization: Replacing 32-bit floating point (FP32) operations with 8-bit integer (INT8) operations directly reduces the operational cost, though FLOPs count may remain similar.
• Knowledge Distillation: Using FLOPs to size the student model appropriately relative to the teacher. The goal is to maximize FLOPs reduction with minimal accuracy loss.

Parameters are the learnable weights and biases within a neural network that are adjusted during training. They represent the model's "memory" and directly influence its capacity and the number of FLOPs required for inference.

• Relationship to FLOPs: The total parameter count and the architecture's connectivity pattern (e.g., dense vs. sparse) are primary determinants of the FLOPs calculation. More parameters generally, but not always, lead to higher FLOPs.
• Example: A model with 7 billion parameters will have a fundamentally different FLOP profile than one with 70 million parameters, even for the same input size.

Inference Latency is the real-world time delay, measured in milliseconds, between submitting an input to a trained model and receiving its output. While FLOPs measure theoretical computational cost, latency is the experienced performance.

• Key Drivers: Latency is determined by FLOPs, memory bandwidth (how quickly weights can be accessed), hardware parallelism (e.g., GPU cores), and software optimization.
• Critical Metric: For production systems, P95 or P99 latency percentiles are often more critical service-level objectives (SLOs) than raw FLOPs, as they directly impact user experience.

Throughput measures the number of inferences a system can process per second (e.g., inferences/sec). It is the inverse of latency under optimal batching conditions and is crucial for high-volume serving.

• FLOPs Efficiency: High-throughput systems aim to maximize the utilization of available FLOP/s (FLOPS per second) on the hardware. Techniques like continuous batching dynamically group requests to keep computational units saturated.
• Economic Impact: Throughput directly correlates with the cost-per-inference, making it a key metric for infrastructure cost control alongside FLOPs.

A Multiply-Accumulate (MAC) operation is a fundamental compute primitive that performs a = a + (b * c). It is the core operation in matrix multiplications and convolutions.

• Relationship to FLOPs: One MAC is often counted as two FLOPs (one multiplication and one addition). However, some literature uses MACs and FLOPs interchangeably, so context is essential.
• Hardware Design: Modern AI accelerators (TPUs, NPUs) are often benchmarked on their peak MAC/s or TOPS (Tera Operations Per Second) capacity, which aligns directly with FLOP-based model analysis.

Model Compression is a suite of techniques aimed at reducing a model's computational footprint (FLOPs) and memory requirements (parameters) for efficient deployment.

• Key Techniques:
  • Pruning: Removing insignificant weights or neurons, reducing active FLOPs.
  • Quantization: Representing weights and activations with lower precision (e.g., from 32-bit to 8-bit), which reduces memory bandwidth and can enable faster low-precision FLOPs on supported hardware.
  • Knowledge Distillation: Training a smaller "student" model to mimic a larger "teacher," achieving lower FLOPs with minimal performance loss.

FLOPS (Floating Point Operations Per Second) is a measure of a hardware system's computational throughput, not to be confused with FLOPs (the count of operations).

• Hardware Benchmark: It quantifies the peak theoretical performance of a CPU, GPU, or AI accelerator (e.g., 100 TFLOPS = 100 trillion FLOPs per second).
• Practical Performance: The actual achieved FLOPS when running a model is its compute utilization. A model requiring 1 TFLOP (the count) running on a 100 TFLOPS chip in 0.02 seconds achieves 50 TFLOPS of utilization (50% efficiency).