Feature Pyramid Network (FPN)

The bottom-up pathway is the standard feedforward computation of a convolutional backbone network (e.g., ResNet). It produces a feature hierarchy with an inherent multi-scale, pyramidal shape. As the network deepens, spatial resolution decreases while semantic strength increases. FPN typically selects the output of the last layer of each stage (a set of layers producing feature maps of the same spatial size) as the set of reference feature maps for pyramid construction.

The top-down pathway hallucinates higher-resolution features by upsampling spatially coarser, but semantically stronger, feature maps from higher pyramid levels. These upsampled maps are then merged with the corresponding feature maps from the bottom-up pathway via lateral connections. Each lateral connection consists of a 1x1 convolutional layer that reduces the channel dimensions of the bottom-up map to a common feature dimension (e.g., 256 channels), enabling element-wise addition with the upsampled top-down map. This fusion combines high-resolution, low-semantic features with low-resolution, high-semantic features.

The output of the FPN is a set of feature maps at multiple scales (e.g., P2, P3, P4, P5), each with the same rich semantic value due to the top-down pathway. Crucially, all levels share a uniform channel depth (e.g., 256-d), simplifying the attachment of detection heads. These maps are used to detect objects at different scales:

• Higher-resolution maps (e.g., P2) are responsible for detecting small objects.
• Lower-resolution maps (e.g., P5) are responsible for detecting large objects. This design replaces the practice of using a single, deep feature map for all detections.

FPN introduces minimal computational overhead. The 1x1 convolutions in the lateral connections and the 2x nearest-neighbor upsampling are low-cost operations. The entire pyramid is constructed once per image, and all levels can be computed in parallel after the backbone pass. This makes FPN significantly more efficient than traditional methods like image pyramids, which require running the detector multiple times on resized versions of the input image, leading to a linear increase in inference time.

In frameworks like Faster R-CNN, FPN is integrated as the feature extractor for the Region Proposal Network (RPN) and the Region of Interest (RoI) pooling head. The RPN places anchor boxes of a single scale on each pyramid level, creating a scale-invariant anchor strategy. For the detection head, RoIs are assigned to a specific pyramid level based on their size (e.g., k = floor(4 + log2(sqrt(area)/224))), ensuring that all RoIs are processed by features of an appropriate receptive field. This dramatically improves Average Precision (AP), especially for small objects.

While designed for object detection, the FPN architecture has become a versatile building block for other dense prediction tasks requiring multi-scale reasoning:

• Instance Segmentation: Used in Mask R-CNN to generate high-quality masks by applying the FPN to the mask prediction branch.
• Semantic Segmentation: Adapted in architectures like U-Net and DeepLab for refining segmentation boundaries by fusing multi-scale context.
• Keypoint Detection: Provides robust features for locating fine-grained structures, such as human pose estimation. Its principle of fusing low-level and high-level features is a foundational concept in modern computer vision.

A multi-scale feature map is a set of feature representations extracted from a single input image at different spatial resolutions. Lower-resolution maps capture high-level semantic context, while higher-resolution maps retain fine-grained spatial details. FPNs are explicitly designed to generate a rich, multi-scale pyramid from a single input pass, enabling a detector to identify objects of vastly different sizes efficiently.

• Semantic Hierarchy: Deep layers (low resolution) understand 'car' or 'person'; shallow layers (high resolution) detect edges and textures.
• Pyramid Construction: Traditional methods like image pyramids are computationally expensive. FPNs build the pyramid within the network itself.

The top-down pathway is the core mechanism of an FPN, where semantically strong, low-resolution features are upsampled and merged with higher-resolution features from earlier layers. This process enriches all scales with high-level semantic information.

• Upsampling: Typically done via nearest-neighbor or transposed convolution to increase spatial dimensions.
• Semantic Strengthening: Allows a high-resolution feature map, which originally had weak semantics (e.g., from an early CNN layer like ResNet's conv2), to also represent complex object categories.

Lateral connections are the 1x1 convolutional layers that transform feature maps from the backbone network (e.g., ResNet) before they are merged with the top-down stream. Their purpose is to reduce channel dimensions and align the semantic strength of the bottom-up features for clean integration.

• Channel Alignment: The backbone's feature maps may have 256, 512, 1024, or 2048 channels. Lateral 1x1 convolutions project all to a common feature dimension (e.g., 256).
• Feature Refinement: They act as adapters, ensuring the spatially rich but semantically weak features are prepared for fusion with the top-down context.

Feature fusion in an FPN refers to the element-wise operation (typically addition) that combines the upsampled top-down feature map with the corresponding laterally connected bottom-up map. This creates the final, semantically and spatially rich feature maps for the pyramid.

• Additive Fusion: The most common method, where features are summed. Alternative methods include concatenation or attention-based weighting.
• Output Pyramid: The result is a set of feature maps (e.g., P2, P3, P4, P5) at different strides (e.g., 4, 8, 16, 32 pixels) relative to the input image.

The backbone network (or encoder) is a standard convolutional neural network, like ResNet or VGG, used as the primary feature extractor in an FPN. The FPN architecture attaches itself to the hierarchical feature maps produced by this backbone.

• Bottom-Up Pathway: The forward pass of the backbone itself constitutes the bottom-up pathway of the FPN.
• Feature Hierarchy: Key activations are taken from specific stages of the backbone (e.g., the output of ResNet's conv2, conv3, conv4, conv5 blocks) to serve as the foundation for the pyramid.

FPN is not a standalone detector but a feature extractor integrated into major two-stage and one-stage detection frameworks, dramatically improving their performance on multi-scale objects.

• Two-Stage Detectors: Faster R-CNN with FPN uses the pyramid features (P2-P6) as the input to the Region Proposal Network (RPN) and for RoI pooling.
• One-Stage Detectors: RetinaNet places classification and regression subnetworks on each level of the FPN output.
• Instance Segmentation: Mask R-CNN leverages FPN to generate higher-quality RoI features for mask prediction.