Fire Module

The first stage of the fire module is the squeeze layer, composed exclusively of 1x1 convolutional filters. This layer's primary function is to reduce the channel dimension of the input feature map. By compressing information across channels before applying more expensive spatial filters, it acts as a computational bottleneck, significantly lowering the number of parameters and FLOPs. For example, reducing 128 input channels to 16 before a 3x3 convolution cuts the parameters for that subsequent layer by nearly 90%.

Following the squeeze layer, the expand layer increases the channel count again using a mix of filter sizes:

• 1x1 convolutions efficiently increase channel depth.
• 3x3 convolutions capture broader spatial patterns and preserve the receptive field. The outputs from these parallel convolutional branches are concatenated along the channel dimension. This design increases representational capacity after the initial squeeze, allowing the network to learn a richer set of features from the compressed representation.

The fire module's genius lies in its trade-off management. A standard 3x3 convolution on many channels is parameter-heavy. By preceding it with a 1x1 squeeze, the module drastically reduces parameters for the 3x3 operation. Crucially, the subsequent 3x3 convolution in the expand layer maintains the same spatial receptive field as a standard network layer. This allows the model to capture complex spatial features without the proportional computational cost, a critical consideration for memory-constrained microcontrollers.

In the SqueezeNet architecture, fire modules are not used in isolation. They are stacked sequentially and sometimes in parallel paths. Designers can adjust three hyperparameters per module:

• s1x1: The number of 1x1 filters in the squeeze layer.
• e1x1: The number of 1x1 filters in the expand layer.
• e3x3: The number of 3x3 filters in the expand layer. Gradually increasing (e1x1 + e3x3) relative to s1x1 across the network builds complexity. Max-pooling layers are strategically placed after some fire modules to downsample spatial dimensions.

The fire module differs from common building blocks:

• vs. Standard Convolution: A single 3x3 conv layer with C input and C output channels has ~9C² parameters. A fire module (squeeze to C/4, then expand) has ~(C*(C/4) + (C/4)33*(C/2) + (C/4)11*(C/2)) ≈ 1.75C² parameters—an ~80% reduction.
• vs. ResNet Bottleneck: A ResNet bottleneck (1x1, 3x3, 1x1) also reduces then expands channels, but its middle 3x3 layer operates on the reduced dimension. The fire module's innovation is the parallel mixed filter expansion after the squeeze, offering a different efficiency profile.

The fire module's principles inspired later designs for embedded ML:

• The concept of squeezing channels before expensive ops is seen in MobileNetV2's inverted residual block, which expands, applies depthwise conv (a spatial filter), then projects back.
• The use of parallel convolutional branches of different sizes influenced Inception modules and their descendants.
• Its focus on parameter count as a primary constraint directly paved the way for neural architecture search (NAS) targeting microcontroller-sized models, where every kilobyte matters.

SqueezeNet is the parent architecture that introduced the Fire Module. It achieves AlexNet-level accuracy on ImageNet with 50x fewer parameters by strategically using 1x1 convolutions to reduce channel counts before expensive 3x3 operations. Its design principles prioritize a small model footprint, making it a foundational architecture for embedded deployment.

• Core Strategy: Replace 3x3 filters with 1x1 filters where possible.
• Architecture: A cascade of Fire Modules and occasional pooling layers.
• Impact: Demonstrated that drastic parameter reduction is possible without catastrophic accuracy loss, paving the way for later mobile-optimized networks.

This is a factorized convolution that decomposes a standard convolution into a depthwise convolution (applying a single filter per input channel) followed by a pointwise convolution (a 1x1 convolution to combine outputs). It is the foundational operation in MobileNet.

• Efficiency Gain: Drastically reduces computations and parameters compared to standard convolution.
• Relation to Fire Module: While the Fire Module uses standard 3x3 convs in its expand layer, depthwise separable convolution offers an even more efficient alternative for spatial filtering, often used in later architectures.

Introduced in MobileNetV2, this block expands the number of channels with a lightweight 1x1 convolution, applies a depthwise convolution for spatial filtering, then projects back to a smaller number of channels. It uses a linear bottleneck (no non-linearity) on the narrow layer to prevent information loss.

• Design Philosophy: Opposite of a traditional residual block; it expands then compresses.
• Contrast with Fire Module: The Fire Module compresses (squeezes) first, then expands. Both are designed for efficiency but employ different expansion/compression strategies.

A structural component that uses 1x1 convolutions to first reduce (compress) and then expand the number of channels. This limits the computational cost of subsequent, more expensive convolutions (e.g., 3x3). Popularized by ResNet for building deep networks efficiently.

• Primary Function: To reduce dimensionality before a costly operation, then restore it.
• Relation to Fire Module: The squeeze layer of a Fire Module is a form of bottleneck, compressing channels before the expand layer's 3x3 convolutions.

A 1x1 convolution that operates across all input channels to combine or project them into a new channel space. It is the primary operation for channel mixing with minimal spatial processing.

• Key Role: Used for channel-wise communication and dimensionality adjustment.
• Ubiquity: Forms the entire squeeze layer and half of the expand layer in a Fire Module. It is also the second step in a depthwise separable convolution and the expansion/projection layers in inverted residual blocks.

A variant where input channels are divided into separate groups, and a separate convolutional filter is applied to each group. This reduces parameters and computations but limits cross-channel interaction.

• Efficiency Trade-off: Increases speed at the cost of restricted feature mixing.
• Architectural Use: Extensively used in ShuffleNet and ResNeXt. The Fire Module does not use grouped convolution; its 3x3 convs in the expand layer are standard, allowing full cross-channel mixing within that layer's input.