Faster R-CNN

The core innovation of Faster R-CNN is the Region Proposal Network, a fully convolutional network that simultaneously predicts object bounds and objectness scores at each sliding window position. Unlike predecessors (R-CNN, Fast R-CNN) that relied on external proposal algorithms like Selective Search, the RPN shares full-image convolutional features with the detection network.

• Anchor Boxes: At each sliding position, the RPN predicts multiple region proposals using pre-defined anchor boxes of varying scales and aspect ratios (e.g., 128², 256², 512² pixels with ratios 1:1, 1:2, 2:1)
• Two sibling output layers: A classification layer predicting object vs. background, and a regression layer refining anchor coordinates
• Training: Uses a multi-task loss combining classification loss (binary cross-entropy) and regression loss (smooth L1) for anchor refinement
• Efficiency gain: Eliminates the computational bottleneck of external proposal generation, reducing proposal time from ~2 seconds (Selective Search) to ~10 milliseconds

Faster R-CNN operates in two sequential stages, separating the tasks of where to look from what is there for superior accuracy.

• Stage 1 — Region Proposal: The RPN scans the feature map and outputs a sparse set of rectangular region proposals, each with an objectness score. Non-Maximum Suppression (NMS) prunes redundant proposals, typically keeping the top 300-2000 regions
• Stage 2 — Detection: Each proposal is processed through RoI Pooling to extract a fixed-size feature vector, which is then fed into fully connected layers for final classification and bounding box regression refinement
• End-to-End Training: The entire pipeline is trained jointly using a 4-step alternating optimization strategy, where RPN and detector networks share convolutional layers
• Medical relevance: This two-stage design excels at detecting small, subtle lesions where high recall is critical, as the RPN can be tuned to propose many candidate regions without overwhelming the downstream classifier

A defining efficiency feature is the shared convolutional feature map used by both the RPN and the detection head. A single forward pass through a backbone network (e.g., VGG-16, ResNet-50) generates a feature map that serves both stages.

• Feature Map: The output of the final convolutional layer (e.g., ~60×40 spatial grid for a 1000×600 input with VGG-16) encodes high-level visual features
• RPN input: A small 3×3 spatial window slides over this feature map, mapping each location to a lower-dimensional feature vector (e.g., 256-d or 512-d)
• Detection head input: The same feature map is cropped via RoI Pooling for each proposed region
• Benefit: Eliminates redundant computation — without sharing, each proposal would require a separate CNN forward pass (as in original R-CNN)
• Medical adaptation: Backbones pre-trained on ImageNet are commonly fine-tuned; for CT scans, the first layer is often modified to accept single-channel input and use Hounsfield Unit (HU) normalization

Faster R-CNN is trained with a multi-task loss that jointly optimizes classification and localization for both the RPN and the final detection head.

• RPN Loss: Combines binary classification loss (object vs. background over 256 sampled anchors) with smooth L1 regression loss for anchor coordinate refinement, balanced with a λ parameter (typically λ=10)
• Detection Loss: Combines multi-class cross-entropy loss (over N+1 classes including background) with class-specific bounding box regression loss
• Smooth L1 Loss: Used for bounding box regression because it is less sensitive to outliers than L2 loss — defined as 0.5x² for |x| < 1 and |x| - 0.5 otherwise
• Hard Negative Mining: During RPN training, anchors are sampled with a 1:1 ratio of positive to negative examples to address class imbalance
• Medical relevance: For lesion detection with extreme foreground-background imbalance, Focal Loss can replace the standard cross-entropy loss to down-weight easy negative examples

After the RPN proposes regions, RoI Pooling converts each variable-sized region proposal into a fixed-size feature map (e.g., 7×7) for the fully connected detection layers.

• Operation: Divides each RoI into a fixed spatial grid (e.g., 7×7 bins) and performs max pooling within each bin
• Quantization: RoI Pooling introduces two quantization steps — mapping proposal coordinates to the feature map grid, and dividing into bins — causing misalignment between the RoI and extracted features
• RoI Align: A successor operation that uses bilinear interpolation at four regularly sampled locations per bin, avoiding quantization and preserving exact spatial locations. This is critical for mask prediction in Mask R-CNN and for precise small-object localization
• Medical relevance: For detecting micro-calcifications or small nodules (< 32×32 pixels), RoI Align's precision-preserving design significantly improves localization accuracy compared to standard RoI Pooling

While the original Faster R-CNN operates on a single-scale feature map, modern implementations integrate a Feature Pyramid Network (FPN) to build a multi-scale feature hierarchy, dramatically improving detection of objects at vastly different sizes.

• FPN Architecture: A bottom-up pathway (standard ResNet), a top-down pathway with lateral connections that upsamples semantically strong but spatially coarse feature maps, producing feature pyramids at multiple scales (e.g., P2 through P6)
• Scale-Aware RPN: Anchors are assigned to different pyramid levels based on their size — small anchors to high-resolution P2, large anchors to coarse P6
• Multi-Scale RoI Pooling: Each proposal is routed to the appropriate pyramid level for feature extraction based on its size: level k = ⌊k₀ + log₂(√wh / 224)⌋
• Medical impact: Essential for simultaneously detecting small lesions (e.g., 16×16 pixel lung nodules) and large anatomical structures (e.g., whole-organ segmentations) within the same scan
• Performance: FPN-equipped Faster R-CNN achieves significant gains in small object detection without additional computational cost

The defining innovation of Faster R-CNN. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each sliding-window position. It shares convolutional layers with the downstream detection network, eliminating the computational bottleneck of external proposal algorithms like Selective Search. The RPN uses anchor boxes at multiple scales and aspect ratios as reference templates, regressing offsets to refine proposals. For medical imaging, the RPN can be tuned to propose regions at scales appropriate for small lesions or large anatomical structures.

A quantization-free feature extraction operation that preserves exact spatial locations. Unlike its predecessor RoI Pooling, which introduced misalignments through harsh quantization, RoI Align uses bilinear interpolation to compute the exact values of input features at regularly sampled points. This pixel-level precision is critical for medical image segmentation tasks where accurate boundary delineation of tumors or lesions directly impacts clinical measurements. RoI Align was introduced in Mask R-CNN and is essential for downstream mask prediction heads.

A post-processing algorithm that eliminates redundant, overlapping bounding boxes for the same object. NMS operates by:

• Sorting all detections by confidence score
• Selecting the highest-scoring box and suppressing all others with IoU above a threshold
• Repeating until no boxes remain In radiology, aggressive NMS can suppress true adjacent abnormalities. Soft-NMS decays scores rather than eliminating boxes, improving recall for clustered findings like micro-calcifications.

A feature extractor backbone extension that builds a multi-scale, pyramidal hierarchy of feature maps. FPN combines semantically strong features from deep layers with spatially precise features from shallow layers via top-down pathways and lateral connections. This architecture enables Faster R-CNN to detect objects at vastly different sizes—from small lung nodules (a few pixels) to large organomegaly findings—without sacrificing speed. FPN is the standard backbone for modern two-stage detectors in medical imaging.

The standard evaluation metric for object detection tasks. mAP calculates the area under the precision-recall curve for each class and averages across all classes. Key parameters:

• IoU thresholds: Typically evaluated at 0.50, 0.75, or averaged across 0.50:0.95
• COCO-style mAP: Averages over 10 IoU thresholds from 0.50 to 0.95 For medical imaging, FROC (Free-Response ROC) is often preferred as it plots sensitivity against false positives per image, better reflecting clinical workflow where radiologists review a limited number of marks.

A multi-stage extension of Faster R-CNN that trains a sequence of detectors with progressively increasing IoU thresholds (e.g., 0.5, 0.6, 0.7). Each stage refines the bounding box proposals from the previous stage, producing higher-quality localizations. This architecture addresses the overfitting problem that occurs when a single detector is trained with a high IoU threshold on limited positive samples. Cascade R-CNN is particularly valuable for lesion localization tasks requiring precise boundary delineation for volumetric measurement.