Inverted Residual Block

A linear bottleneck is the final 1x1 convolution layer in the block that projects the expanded features back to a lower-dimensional output. Crucially, it uses a linear activation function (no ReLU). This prevents non-linearities from destroying information in the low-dimensional compressed representation, a key insight from MobileNetV2 research. Without this, repeated ReLU activations in narrow spaces can cause irreversible information loss, crippling the network's capacity.

The block uses a channel expansion ratio (typically 6) to first increase the number of channels using a cheap 1x1 convolution. This creates a wider, higher-dimensional space where depthwise convolution operates more effectively. After the depthwise convolution, a second 1x1 projection convolution compresses the channels back down. This 'expand → filter → compress' flow is the inverse of a classic ResNet bottleneck, which compresses first.

At the heart of the block is a depthwise separable convolution. This factorized operation applies a single convolutional filter per input channel, dramatically reducing computations compared to a standard convolution. It performs spatial filtering without mixing channels. The combination of expansion, depthwise convolution, and projection is far more efficient than a standard 3x3 convolution with the same input/output dimensions.

Like standard ResNet blocks, inverted residual blocks employ a skip connection or identity shortcut. This connection adds the block's input directly to its output, which:

• Mitigates vanishing gradients for deeper networks.
• Enables feature reuse.
• Often only connects blocks where the input and output have the same number of channels (stride=1). When a stride of 2 is used for downsampling, the residual connection is typically omitted.

The inverted design is optimized for memory-constrained inference. The expensive intermediate tensors (the expanded features) exist only within the block and are consumed by the next layer. The network primarily stores the much smaller input/output tensors. This is critical for microcontroller deployment where SRAM is severely limited (often < 512KB). The block minimizes peak memory usage during the graph execution.

Classic ResNet Bottleneck (compress → filter → expand):

• 1x1 conv: Compresses channels.
• 3x3 conv: Filters in low-dim space.
• 1x1 conv: Expands channels.
• High risk of information loss in the narrow middle.

Inverted Residual Block (expand → filter → compress):

• 1x1 conv: Expands channels.
• 3x3 depthwise: Filters in high-dim space.
• 1x1 linear conv: Compresses channels.
• Preserves information by filtering in an expanded space.

The foundational operation upon which the Inverted Residual Block is built. It factorizes a standard convolution into two efficient steps:

• Depthwise Convolution: Applies a single filter per input channel, performing spatial filtering.
• Pointwise Convolution: A 1x1 convolution that mixes the channel outputs from the depthwise step. This decomposition drastically reduces parameters and computations compared to a standard 3x3 convolution, making it essential for embedded vision models.

A critical design insight within the Inverted Residual Block. It refers to the layer where the expanded channels are projected back down, using a linear activation function (i.e., no non-linearity like ReLU).

• Purpose: In low-dimensional spaces, non-linear activation functions like ReLU can destroy information. Using a linear activation in the bottleneck preserves the manifold of interest, maintaining the network's representational power despite severe compression.
• Contrast: This is the inverse of a traditional residual block, which uses non-linearities in the bottleneck.

The general structural concept that the Inverted Residual Block adapts. A bottleneck layer uses 1x1 convolutions to first reduce (compress) the number of channels, applies a heavier operation (like a 3x3 conv), and then expands back.

• Classic Use: Found in ResNet, it reduces FLOPs before expensive convolutions.
• Inverted Adaptation: The Inverted Residual Block reverses this: it first expands with a cheap 1x1 convolution, applies depthwise convolution, then compresses (projects) back. This places the high-dimensional, information-rich representation in the wider, computationally cheap expansion layer.

The 1x1 convolution operation that serves as the workhorse for channel manipulation in efficient blocks.

• In the Inverted Residual Block: It performs two key roles:
  1. Expansion: The first 1x1 conv increases the channel count (e.g., by a factor of 6).
  2. Projection: The final 1x1 conv reduces the channel count back down.
• Efficiency: While still a full connection across channels, its kernel size of 1x1 makes it far less computationally intensive than larger kernels, ideal for mixing and transforming channel information.

The "shortcut" or "skip connection" that the block employs, allowing gradients to flow directly through the network.

• Function: It adds the input of the block directly to its output, mitigating the vanishing gradient problem and enabling the training of very deep networks.
• Inverted Block Specifics: The connection is typically applied around the expanded depthwise convolution. If the input and output dimensions match, it's an identity connection. If they differ (e.g., when downsampling), a linear projection or average pooling is used to match dimensions.