Non-Maximum Suppression (NMS)

NMS operates on a greedy sorting and pruning principle. It first ranks all predicted bounding boxes by their confidence score in descending order. The highest-scoring box is selected as a valid detection. All other boxes with an Intersection over Union (IoU) exceeding a predefined threshold (typically 0.5) are suppressed, as they are considered redundant detections of the same object. This process iterates until no boxes remain.

• Input: A set of bounding boxes with class labels and confidence scores
• Sorting: Descending order by confidence score
• Suppression Criterion: IoU > threshold (e.g., 0.5)
• Output: A sparse set of non-overlapping, high-confidence detections

The IoU threshold is the single most critical hyperparameter in NMS, directly controlling the trade-off between precision and recall. A low threshold (e.g., 0.3) aggressively suppresses boxes, reducing false positives but risking the removal of true adjacent objects (e.g., two distinct but proximate micro-calcifications). A high threshold (e.g., 0.7) is more permissive, preserving true positives but potentially allowing duplicate detections to persist.

• Low Threshold (0.3): High precision, low recall; risks missing clustered objects
• High Threshold (0.7): High recall, low precision; risks duplicate detections
• Clinical Impact: In mammography, an overly aggressive threshold could suppress a small satellite lesion adjacent to a primary mass

Standard NMS applies a hard, binary penalty—boxes are either kept or completely discarded. Soft-NMS introduces a continuous penalty function that decays the confidence scores of overlapping boxes rather than zeroing them out. The decay is proportional to the IoU overlap: a box with 0.9 IoU is heavily penalized, while a box with 0.6 IoU receives a smaller penalty. This prevents the sudden removal of boxes that might represent distinct, nearby objects.

• Linear Penalty: score = score * (1 - IoU)
• Gaussian Penalty: score = score * exp(-IoU² / σ)
• Advantage: Improves recall for crowded scenes without introducing a complex additional network

NMS can be applied in two modes depending on the detection architecture. Class-agnostic NMS suppresses any overlapping boxes regardless of their predicted class, which is computationally simpler but can erroneously suppress a true positive of one class in favor of a higher-confidence false positive of another. Class-specific NMS applies suppression independently within each class channel, preserving detections of different classes that spatially overlap.

• Class-Agnostic: Single pass; risk of inter-class suppression
• Class-Specific: Parallel passes per class; preserves co-located objects of different types
• Example: In chest radiography, a predicted 'nodule' should not suppress an overlapping 'rib fracture' detection

Traditional NMS is a hand-crafted, non-differentiable heuristic that breaks end-to-end training. Modern architectures address this limitation. DETR (DEtection TRansformer) eliminates NMS entirely by using a bipartite matching loss and a fixed set of object queries, enforcing one-to-one predictions. Relation Networks learn a differentiable NMS module that can be trained jointly with the detector, modeling the relationships between detections rather than applying a fixed rule.

• DETR: No NMS required; set-based global loss
• Relation Networks: Learnable, differentiable NMS module
• CenterNet: Anchor-free detection with local maxima extraction replacing NMS

In a Computer-Aided Detection (CADe) pipeline, NMS is the final algorithmic step before results are presented to the radiologist or encoded into a DICOM Structured Report (DICOM SR). The suppressed output ensures that a single lesion generates exactly one mark on the viewing workstation, preventing alert fatigue from duplicate bounding boxes. The final confidence scores attached to each surviving box are used to prioritize findings in the worklist.

• Pre-NMS: Hundreds of overlapping proposals per image
• Post-NMS: A clean set of distinct, actionable findings
• DICOM SR: Surviving boxes are serialized with coordinates and measurements for PACS integration

The geometric metric that serves as the primary gating function for NMS. IoU measures the overlap between two bounding boxes by dividing the area of their intersection by the area of their union.

• IoU Threshold: The critical hyperparameter in NMS—boxes with IoU above this value are considered redundant
• Typical Values: 0.5 for standard detection, 0.3-0.4 for crowded scenes with small objects
• Soft-NMS Variant: Instead of zeroing scores, Soft-NMS applies a Gaussian penalty function that decays confidence proportionally to IoU, preserving nearby detections of distinct objects

A threshold set too high results in duplicate detections; too low suppresses legitimate adjacent objects.

The upstream process that generates the candidate boxes NMS must filter. A regressor network predicts offsets (dx, dy, dw, dh) to refine anchor boxes or initial proposals into precise object boundaries.

• Coordinate Parametrization: Offsets are predicted in log-space for scale invariance and normalized by anchor dimensions
• Smooth L1 Loss: The standard regression loss that combines L1 robustness with L2 smoothness near zero
• Cascade Refinement: Multi-stage detectors apply sequential regression steps, each with increasing IoU thresholds, producing higher-quality boxes that require less aggressive NMS

Poor regression accuracy forces NMS to handle greater spatial uncertainty, increasing the risk of false suppression.

The ranking criterion that determines which boxes survive NMS. Each detection carries a scalar probability representing the model's certainty that the box contains an object of a specific class and is accurately localized.

• Calibration Gap: Raw confidence scores are often poorly calibrated—a 0.9 score may not correspond to 90% probability of correctness
• Class-Aware NMS: Modern implementations apply suppression per-class independently, preventing a high-confidence 'cat' detection from suppressing a lower-confidence 'dog' detection in the same spatial region
• Score Decay in Soft-NMS: Instead of hard elimination, scores are attenuated by a function f(IoU), allowing boxes with initially lower confidence to survive if they are spatially distinct

The confidence score acts as the arbitration mechanism: the highest-scoring box suppresses all overlapping lower-scoring neighbors.

The evaluation metric that directly measures NMS effectiveness. mAP computes the area under the precision-recall curve, averaged across classes and IoU thresholds.

• IoU@[.50:.95]: The COCO standard evaluates mAP at 10 IoU thresholds from 0.50 to 0.95 in 0.05 increments, heavily penalizing poor localization
• NMS Sensitivity: Overly aggressive NMS reduces recall (false negatives increase), while lenient NMS reduces precision (false positives increase)—mAP captures this trade-off
• FROC Alternative: In medical imaging, Free-Response ROC curves plot sensitivity against false positives per image, directly quantifying the clinical burden of NMS failures

Tuning NMS hyperparameters is fundamentally an mAP optimization problem on the validation set.

An end-to-end detection architecture that eliminates the need for NMS entirely. DETR uses a transformer encoder-decoder with a set-based global loss and bipartite matching to produce a fixed-size set of predictions.

• Bipartite Matching: The Hungarian algorithm assigns each ground truth box to exactly one prediction during training, enforcing one-to-one correspondence
• No Post-Processing: The model learns to output non-overlapping detections directly, bypassing the heuristic NMS step
• Limitations: DETR struggles with small objects and requires extremely long training schedules (500+ epochs) to converge

DETR demonstrates that NMS is an architectural artifact of anchor-based and proposal-based detectors, not a fundamental requirement of object detection.

An ensembling alternative to NMS that merges rather than eliminates overlapping boxes. WBF combines predictions from multiple models by averaging their coordinates and confidence scores.

• Fusion Algorithm: All boxes are sorted by confidence, then iteratively fused if their IoU exceeds a threshold—unlike NMS, no box is discarded
• Coordinate Averaging: Fused box coordinates are weighted by confidence scores: boxes with higher confidence contribute more to the final position
• Ensemble Advantage: WBF consistently outperforms NMS-based ensembling in competitions because it preserves the geometric consensus of multiple models

WBF is particularly valuable in medical imaging ensembles where combining the spatial wisdom of diverse architectures improves lesion localization accuracy.