ResNet

The fundamental unit of a ResNet is the residual block. Instead of a stack of layers learning a desired underlying mapping H(x), the block is designed to learn a residual function F(x) = H(x) - x. The original input x is passed forward via an identity skip connection and added to the output of the block's layers: Output = F(x) + x. This structure mitigates the vanishing gradient problem by providing a direct, unmodified path for gradients to flow backward through the network during training.

• Identity Mapping: The skip connection performs an element-wise addition, not a concatenation.
• Ease of Optimization: The network can learn to drive the residual F(x) towards zero, making it easier to train layers as the network depth increases.

For very deep networks (e.g., ResNet-50, 101, 152), a bottleneck block is used to improve computational efficiency. This design reduces the number of parameters and FLOPs. A standard bottleneck block for ResNet-50/101/152 has the structure:

1. 1x1 Convolution: Reduces the channel dimensionality (e.g., from 256 to 64).
2. 3x3 Convolution: Performs spatial feature extraction on the reduced channel set.
3. 1x1 Convolution: Expands the channel dimensionality back up (e.g., from 64 to 256).

This 1x1 -> 3x3 -> 1x1 pattern creates a computational bottleneck, allowing for deeper networks without a prohibitive increase in cost. The skip connection connects the input to the output of this three-layer stack.

When the spatial dimensions are halved (e.g., from 56x56 to 28x28) and the number of channels is doubled, the identity mapping from the input x cannot be directly added to the output F(x) due to dimension mismatch. ResNet handles this in two ways:

• Option A: The shortcut still performs an identity mapping, but extra zero entries are padded to increase channels. This adds no extra parameters.
• Option B (Projection Shortcut): A 1x1 convolutional layer with a stride of 2 is used in the shortcut path. This layer projects the input to the correct dimensions (channels and spatial size) and is parameterized. This is the more common approach in modern implementations as it provides a learned projection.

These mechanisms allow the residual learning framework to be applied consistently across the entire network, even when changing feature map dimensions.

Prior architectures like VGG and AlexNet used one or more fully connected (dense) layers at the end of the convolutional feature extractor for classification. ResNet popularized the use of Global Average Pooling (GAP) as the final layer before the classification head. Instead of flattening the feature map, GAP takes the spatial average of each channel in the final convolutional feature map.

• Reduces Parameters: Eliminates the need for massive dense layers, drastically cutting the number of parameters.
• Improves Generalization: Acts as a regularizer, reducing overfitting.
• Input Size Flexibility: Makes the network more robust to varying input spatial sizes, as the pooling operation is adaptive.

The output of GAP is a single vector per channel, which is then fed into a single, small fully connected layer for final class prediction.

ResNet architectures integrate Batch Normalization (BN) after every convolutional layer and before the activation function (typically ReLU). This was a key design choice for stable training of very deep networks.

• Standard Order: The common pattern within a residual block is Conv -> BN -> ReLU.
• Stabilizes Activations: BN normalizes the activations of the previous layer, reducing internal covariate shift. This allows for higher learning rates and provides a mild regularization effect.
• Critical for Depth: The combination of BN and residual connections was instrumental in enabling the successful training of networks with over 100 layers. The original ResNet paper placed BN after the addition of the skip connection, but common practice now places it inside the residual path before the addition.

The ResNet design spawned a family of models defined by their depth, measured in the number of weighted layers. The key variants demonstrate the scalability of the architecture:

• ResNet-18/34: Use basic blocks containing two 3x3 convolutional layers.
• ResNet-50/101/152: Use bottleneck blocks (1x1, 3x3, 1x1 convs) for efficiency.
• Performance Scaling: Deeper models (ResNet-152) generally achieve higher accuracy on benchmarks like ImageNet, showing that the residual framework effectively counteracts degradation.

Later Evolutions:

• ResNeXt: Introduces a cardinality dimension, using grouped convolutions within the bottleneck block for a multi-branch design.
• Wide ResNet: Increases the width (number of filters) per layer instead of depth, often yielding better performance with fewer layers.
• ResNet-D: A refined version that tweaks the downsampling block and stem for improved accuracy.

A residual connection, or skip connection, is the core innovation of ResNet. It creates a direct pathway for the gradient to flow through the network by adding the input of a block to its output via an identity mapping. This formulation, expressed as (F(x) + x), allows the network to learn residual functions (F(x)) relative to the input, making it easier to optimize very deep architectures by mitigating the vanishing gradient problem.

The vanishing gradient problem is a challenge in training deep neural networks where gradients calculated during backpropagation become exponentially small as they are propagated backward through many layers. This prevents weights in the earlier layers from updating effectively, stalling learning. ResNet directly addresses this by providing shortcut paths (residual connections) that allow gradients to flow unimpeded, enabling the stable training of networks with hundreds or even thousands of layers.

Batch Normalization is a technique that normalizes the inputs to a layer for each mini-batch during training. It stabilizes learning by reducing internal covariate shift—the change in the distribution of layer inputs. While not invented for ResNet, it is a critical companion technology. In the original ResNet paper, batch normalization is applied after each convolution and before activation, which works synergistically with residual connections to enable the training of extremely deep networks.

DenseNet (Densely Connected Convolutional Network) is an architecture that extends the concept of shortcut connections. Instead of adding the input to the output of a block, DenseNet concatenates the feature maps of all preceding layers to the inputs of all subsequent layers. This creates dense connectivity, encouraging feature reuse, reducing the number of parameters, and improving gradient flow. It represents an evolution of the connectivity principles first popularized by ResNet.

Highway Networks are a precursor to ResNet that introduced adaptive gated shortcut connections. They use learned gating mechanisms to regulate how much information flows through a transformation path versus a carry (skip) path. While ResNet simplified this concept by using fixed, ungated identity shortcuts (with optional projection for dimension matching), Highway Networks demonstrated the feasibility of training very deep networks with bypass pathways, directly inspiring the ResNet architecture.

ResNet is defined by a family of models with different depths, primarily distinguished by the number of layers:

• ResNet-18/34: Use basic blocks with two 3x3 convolutions.
• ResNet-50/101/152: Use bottleneck blocks with a 1x1 convolution for dimensionality reduction, a 3x3 convolution, and a 1x1 convolution for restoration. This design increases depth while being computationally more efficient. The number (e.g., 50) refers to the count of weight layers. These variants provide a clear trade-off between accuracy, computational cost, and model size for different applications.