Compound Scaling

The compound coefficient (φ) is a single, user-specified scaling constant that uniformly controls the increase in network depth, width, and input resolution. It defines the scaling relationship:

• Depth (d): Number of layers. Scales as α^φ
• Width (w): Number of channels. Scales as β^φ
• Resolution (r): Input image dimensions. Scales as γ^φ The constants α, β, γ are small coefficients (≈1.2) determined via a small grid search on a baseline model, subject to the constraint α · β² · γ² ≈ 2. This ensures total FLOPS increase by roughly 2^φ.

Traditional scaling methods often focus on a single dimension:

• Depth Scaling: Improves representational capacity but leads to vanishing gradients.
• Width Scaling: Improves feature granularity but struggles with higher-level features.
• Resolution Scaling: Captures finer patterns but increases computational cost quadratically. Compound scaling posits that balancing all three dimensions is crucial. For example, higher-resolution images require a deeper network to have a larger receptive field and a wider network to capture more fine-grained patterns. The method prevents the diminishing returns observed when scaling any single dimension in isolation.

The process begins with a baseline network (e.g., EfficientNet-B0) designed via Neural Architecture Search (NAS) for optimal performance under a fixed resource budget. A small grid search is then performed on this baseline to find the optimal values for the base scaling constants α, β, γ. This search maximizes validation accuracy under the constraint that increasing any dimension by these constants increases FLOPS by a similar amount. Once determined, these constants are fixed for the entire model family (EfficientNet-B0 to B7).

Scaling is performed under a target computational budget, typically total FLOPS. The relationship is formalized as: Total FLOPS ≈ (α · β² · γ²)^φ Because depth scales computations linearly, width scales quadratically (per convolutional layer), and resolution scales quadratically (for the input tensor). The constraint α · β² · γ² ≈ 2 means that for each step increase in the compound coefficient φ, the computational cost approximately doubles. This provides a predictable and controlled method for generating a family of models (e.g., B0-B7) with exponentially increasing capacity and cost.

Compound scaling is applied uniformly per-stage across the network's architecture. A modern CNN (like MobileNetV2 or EfficientNet) is composed of multiple stages with different convolutional blocks. The scaling rules are applied to all layers within a given stage consistently. For instance, when scaling width, the number of output channels for all convolutional layers in a stage is multiplied by β^φ. This preserves the designed architectural ratios and block structures discovered by NAS, ensuring scaled-up models retain the efficiency properties of the baseline.

The efficacy of compound scaling was validated on ImageNet and transfer learning datasets. Key results include:

• EfficientNet-B7 achieved state-of-the-art 84.4% top-1 ImageNet accuracy with 8.4x fewer parameters and 6.1x lower FLOPS than the previous best model (GPipe).
• It demonstrated superior performance compared to single-dimension scaling, which plateaued in accuracy gains.
• The scaled models showed strong transfer learning performance on CIFAR-100, Flowers, and other benchmarks, proving the generalized effectiveness of the discovered scaling principles beyond the training task.

The core innovation of compound scaling is the use of a single, user-defined compound coefficient (φ) to uniformly scale three network dimensions:

• Depth (d): Number of layers. Scaled as d = α^φ.
• Width (w): Number of channels per layer. Scaled as w = β^φ.
• Resolution (r): Input image height/width. Scaled as r = γ^φ. The constants α, β, γ are determined via a small grid search on the base model to maximize accuracy per floating-point operation. This single-knob control ensures balanced scaling, preventing the diminishing returns of scaling any single dimension in isolation.

The compound scaling rule is bidirectional. While it scales up for data center performance, it can also scale down to create highly efficient models for edge deployment. By using a compound coefficient φ < 0, developers can generate a family of smaller models from a single base architecture. This ensures consistent architectural principles across a product line, from a microcontroller version to a cloud API, simplifying maintenance and deployment pipelines. It provides a systematic alternative to manually designing separate lightweight architectures like MobileNet or ShuffleNet.

Compound scaling is typically applied after a high-performance base architecture is found via Neural Architecture Search (NAS). The workflow is a two-stage process:

1. NAS Stage: Use NAS (e.g., using the Accuracy × [FLOPS]^T reward) to discover the optimal baseline cell structure and macro-architecture (EfficientNet-B0).
2. Scaling Stage: Apply the compound scaling rule with increasing φ to create larger models (B1-B7). This decoupling is crucial: NAS finds an optimal shape, compound scaling finds the optimal size. It is more efficient than running NAS for every target compute budget.

While pioneered for CNNs, the principle of balanced multi-dimensional scaling has been adapted for other architectures:

• Vision Transformers (ViTs): Scaling dimensions include depth (number of transformer blocks), width (embedding dimension), and the number of attention heads, alongside input resolution.
• Large Language Models (LLMs): Scaling laws often consider model size (parameters), dataset size, and compute budget. While not an exact analog, the concept of joint, balanced scaling is a key research theme. The core lesson is that scaling any single dimension (e.g., just depth) leads to suboptimal performance; dimensions must be scaled in concert.

Compound scaling solves the common but inefficient practice of arbitrary scaling. Key differentiators:

• ResNet Scaling: Typically scaled only in depth (e.g., ResNet-50 to ResNet-152), leading to saturation in accuracy gains.
• Manual Width/Resolution Scaling: Ad-hoc increases in width or resolution are common but unprincipled. Compound scaling provides a quantitative, data-driven formula. A model scaled via compound coefficient φ will consistently achieve higher accuracy than one scaled by the same FLOPS budget using a naive single-dimension approach, as proven in the original EfficientNet paper.