Feature Pyramid Network (FPN)

FPNs construct a pyramidal hierarchy of feature maps where each level captures information at a different spatial resolution. Unlike image pyramids that process the input at multiple scales, FPNs build the pyramid within the network itself using the inherent multi-scale structure of convolutional backbones.

• Bottom-up pathway: Standard feed-forward computation producing feature maps at progressively lower resolutions
• Top-down pathway: Upsamples semantically strong but spatially coarse features from higher pyramid levels
• Lateral connections: 1×1 convolutions that merge top-down upsampled features with corresponding bottom-up features of the same spatial size

Standard convolutional networks suffer from a semantic gap: shallow layers have high spatial resolution but weak semantics, while deep layers have strong semantics but coarse spatial resolution. FPNs resolve this by creating a feature pyramid where every level contains strong semantic information.

• Deep features with rich categorical semantics are propagated downward via upsampling
• Shallow features with precise localization information are refined via lateral connections
• Each pyramid level independently predicts objects at its corresponding scale
• Small objects are detected on high-resolution, semantically strong feature maps

FPNs assign objects to pyramid levels based on their size in pixels, ensuring each object is detected at the appropriate resolution. This scale-aware assignment is critical in medical imaging where abnormalities range from sub-millimeter micro-calcifications to large organ-spanning tumors.

• Objects are assigned to pyramid level Pₖ based on their bounding box area
• Level formula: k = ⌊k₀ + log₂(√(wh) / input_size)⌋
• Small nodules route to high-resolution P₂ or P₃
• Large masses route to low-resolution P₅ or P₆
• Prevents the network from learning conflicting scale-specific features at a single level

FPNs function as a neck module that can be attached to any convolutional backbone architecture. This modularity allows the pyramid to leverage advances in backbone design without architectural changes.

• Compatible with ResNet, ResNeXt, EfficientNet, and transformer-based backbones
• Operates on the feature hierarchy naturally produced by modern architectures
• Standard output levels: P₂, P₃, P₄, P₅ (optionally P₆, P₇ via strided convolution)
• Each level outputs 256 channels regardless of backbone depth, ensuring consistent downstream head design
• Enables transfer learning by swapping backbones while preserving the detection head

FPNs employ weight-shared detection heads across all pyramid levels. A single set of convolutional predictors slides over every level, dramatically reducing parameters while enforcing scale-invariant feature representations.

• Identical head architecture processes P₂ through P₆
• Forces the network to learn features that generalize across scales
• Reduces overfitting risk compared to level-specific heads
• Common in Faster R-CNN with FPN and RetinaNet architectures
• Heads typically consist of small fully convolutional subnetworks for classification and bounding box regression

Traditional image pyramid approaches process the input at multiple scales independently, multiplying computation by the number of scales. FPNs achieve comparable multi-scale representation with marginal computational overhead.

• Image pyramids: 5 scales = ~5× computation
• FPN: ~1.2× computation over the base backbone
• Top-down pathway adds only lightweight upsampling and 1×1 convolutions
• Lateral connections are parameter-efficient (1×1 convs)
• Enables real-time or near-real-time inference critical for clinical workflows
• Memory overhead is manageable due to shared feature computation in the bottom-up pathway

A fully convolutional network that serves as the attention mechanism for two-stage detectors. The RPN slides over the FPN's multi-scale feature maps and predicts objectness scores and bounding box coordinates at each position.

• Generates class-agnostic region proposals
• Shares convolutional features with the downstream detection head
• FPN provides the RPN with semantically strong features at all scales
• Crucial for detecting lesions of varying sizes in a single pass

A quantization-free feature extraction operation that preserves exact spatial locations when cropping features from FPN levels. Unlike RoI Pooling, RoI Align uses bilinear interpolation to compute input features at sampled points.

• Eliminates the coarse spatial quantization introduced by RoI Pooling
• Essential for pixel-accurate mask prediction in Mask R-CNN
• Works by assigning each Region of Interest to a specific FPN level based on its scale
• Improves small object detection accuracy by preserving fine-grained spatial information

A specialized sub-field addressing the challenge of localizing objects occupying very few pixels—such as micro-calcifications in mammograms or early-stage lung nodules. FPN is a foundational solution to this problem.

• Small objects suffer from low resolution and limited feature representation in deep layers
• FPN's high-resolution, semantically rich shallow pyramid levels dramatically improve recall
• Typical challenges include extreme class imbalance and high false positive rates
• Often combined with hard negative mining to suppress background false alarms

Predefined bounding box templates of varying scales and aspect ratios that serve as reference priors for detection. In an FPN-based system, anchors are assigned to specific pyramid levels based on their size.

• Larger anchors are assigned to deeper, coarser feature maps
• Smaller anchors are assigned to shallower, higher-resolution feature maps
• This scale-aware assignment is a core design principle of FPN
• Anchor-free alternatives like CenterNet bypass this heuristic entirely

A loss function designed to address the extreme foreground-background class imbalance inherent in dense object detection. It reshapes the standard cross-entropy loss to down-weight easy examples.

• Introduced with the RetinaNet single-stage detector
• RetinaNet uses an FPN backbone to generate a dense pyramid of detections
• Focal Loss prevents the overwhelming number of easy background anchors from dominating training
• Enables single-stage detectors to match the accuracy of two-stage counterparts

A post-processing algorithm that eliminates redundant, overlapping bounding boxes for the same object. After an FPN-based detector generates thousands of proposals, NMS filters them to a clean set.

• Sorts detections by confidence score
• Iteratively selects the highest-scoring box and suppresses others with high Intersection over Union (IoU)
• Soft-NMS decays scores instead of hard-suppressing, improving recall for clustered lesions
• Essential for producing a clinically interpretable output without duplicate marks