Squeeze-and-Excitation Block

The Squeeze operation aggregates global spatial information from a feature map into a compact channel descriptor. It typically uses Global Average Pooling (GAP) to reduce each 2D feature channel (H x W) to a single scalar value. This creates a vector whose length equals the number of input channels (C), capturing the global distribution of channel-wise responses and serving as an efficient summary of the 'excitement' across the spatial domain.

The Excitation operation uses the squeezed descriptor to generate a set of channel-wise weighting factors. This is implemented by a simple, parameter-efficient self-gating mechanism—typically a small multi-layer perceptron (MLP) with one hidden layer and a sigmoid activation. The network learns to model non-linear, non-mutually-exclusive dependencies between channels, outputting a vector of attention weights (between 0 and 1) that selectively emphasize informative features and suppress less useful ones.

SE blocks are designed as plug-and-play modules that can be inserted into existing architectures (e.g., ResNet, MobileNet) after any convolutional block. Despite their power, they add minimal computational overhead. The primary cost comes from the fully-connected layers in the excitation step, but this is negligible compared to the cost of the base convolutions. For a block with C channels and a reduction ratio r, the extra parameters are only 2*C^2/r. This makes them highly suitable for embedded neural network architectures where a small boost in representational power is critical.

The SE block is a foundational form of channel attention. It differs from spatial attention mechanisms (which focus on 'where' to attend) by focusing on 'what' is important at the channel level. This concept of modeling interdependencies between features has been highly influential, paving the way for more sophisticated attention modules in both convolutional networks and vision transformers. It demonstrates that significant performance gains can be achieved by explicitly modeling feature relationships, not just increasing depth or width.

While powerful, the standard SE block's fully-connected layers can be costly for microcontroller inference. For TinyML deployment, several optimizations are common:

• Reducing the reduction ratio (r) to its minimum (e.g., r=2 or r=4) to cut parameters.
• Replacing the MLP with a simpler gating function (e.g., using depthwise convolution).
• Removing SE blocks entirely in ultra-efficient variants like EfficientNet-Lite, where their non-linearities and extra ops are incompatible with integer-only accelerators. The design trade-off involves balancing the accuracy boost against the added latency and SRAM usage for the excitation weights.

SE blocks are part of a broader toolkit for building efficient networks. Key related concepts include:

• Inverted Residual Blocks (MobileNetV2): Use linear bottlenecks and expansion layers for efficient feature transformation.
• Depthwise Separable Convolution: Factorizes standard convolution to drastically reduce computations, often used in tandem with SE blocks.
• Ghost Modules: Generate more feature maps via cheap linear operations, another method to reduce redundant computations.
• Channel Shuffle (ShuffleNet): Enables cross-group information flow after grouped convolutions. Together, these components enable the co-design of accurate yet frugal models for embedded neural network architectures.

A factorized convolution that decomposes a standard convolution into two efficient operations: a depthwise convolution (applies a single filter per input channel) followed by a pointwise convolution (a 1x1 convolution to combine channel outputs). This decomposition drastically reduces parameters and FLOPs compared to standard convolutions, forming the computational backbone of mobile architectures like MobileNet.

A mobile-optimized building block, first introduced in MobileNetV2, designed to preserve representational power while minimizing memory footprint. Its structure follows an expand-transform-squeeze pattern:

• Expands channels with a lightweight 1x1 convolution.
• Applies a depthwise convolution.
• Projects back to fewer channels using a linear bottleneck. The use of a linear activation in the bottleneck prevents ReLU from destroying information in low-dimensional spaces.

A structural component that uses 1x1 convolutions to first reduce (compress) and then expand the number of channels. By limiting the channel count before expensive 3x3 convolutions, it dramatically reduces computational cost. Popularized by ResNet, it is a key technique for building deep yet efficient networks. In mobile architectures, it often pairs with a linear bottleneck to maintain information flow.

A broader family of mechanisms that, like the SE block, aim to model interdependencies between channels. The core idea is to dynamically recalibrate channel-wise feature responses based on the global context. While the SE block uses global average pooling and fully-connected layers, other variants like ECA-Net use 1D convolutions for a lighter-weight implementation, making them more suitable for TinyML.

A lightweight convolutional block designed to reduce redundant computations. Instead of generating all feature maps via costly convolutions, it:

• Generates a set of intrinsic features with a regular convolution.
• Applies cheap linear operations (like depthwise convolutions) to each intrinsic feature to produce "ghost" features. This approach builds efficient networks with fewer parameters and FLOPs, offering an alternative pathway to efficiency for embedded deployment.

The fundamental building block of the SqueezeNet architecture, designed for extreme parameter reduction. It consists of two layers:

• Squeeze layer: Uses only 1x1 filters to reduce input channels.
• Expand layer: Uses a mix of 1x1 and 3x3 filters to increase channels. This design achieves AlexNet-level accuracy on ImageNet with 50x fewer parameters, demonstrating how architectural innovation can enable deployment in highly constrained environments.