Non-Maximum Suppression (NMS)

The primary function of NMS is to select the single best bounding box from a cluster of overlapping candidate boxes that all likely refer to the same physical object. It does this by:

• Sorting all candidate boxes for a given class by their confidence score (e.g., objectness or class probability).
• Iteratively selecting the highest-scoring box as the 'winner'.
• Suppressing (removing) all other boxes whose Intersection over Union (IoU) with the winner exceeds a predefined threshold (e.g., 0.5).
• Repeating this process on the remaining boxes. This prevents the visual clutter of multiple boxes on a single object.

The Intersection over Union (IoU) threshold is the most critical hyperparameter controlling NMS behavior. It defines the overlap level at which two boxes are considered duplicates.

• Low Threshold (e.g., 0.3): Aggressive suppression. Only very loosely overlapping boxes are kept, potentially missing objects that are close together.
• High Threshold (e.g., 0.7): Permissive suppression. Allows more boxes to remain, which can lead to multiple detections on a single object.
• Typical Default: 0.5 is a common starting point, balancing precision and recall. This value must be tuned for the specific object density and scale in the target application, such as robotics where objects may be in close proximity.

NMS is a greedy algorithm that relies on the assumption that the detection with the highest confidence score is the most accurate localization of the object. The algorithm's steps are:

1. Sort All Detections: Order all bounding box proposals for a specific class by their confidence score, descending.
2. Select and Keep: Take the top-scoring box and add it to the final output list.
3. Compute IoU and Suppress: Calculate the IoU between this selected box and every other box in the list. Remove (suppress) all boxes where IoU > threshold.
4. Iterate: Repeat steps 2-3 on the remaining, unsuppressed boxes until none are left. This sequential, score-dependent process ensures the most confident predictions are preserved.

NMS can be applied in two primary modes, affecting how detections for different object classes are handled:

• Class-Agnostic NMS: Suppression is performed on all boxes regardless of their predicted class label. Boxes of different classes can suppress each other if they overlap significantly, which is often undesirable. This is simpler but less accurate for cluttered scenes.
• Class-Aware NMS (the standard): Suppression is performed independently per class. All boxes for 'person' are processed together, separate from boxes for 'car'. This prevents a high-confidence 'car' box from suppressing a valid, overlapping 'person' box, preserving detections in multi-object scenarios common in robotic perception.

Standard Greedy NMS has known limitations that have led to the development of advanced variants:

• Limitation: Occlusion & Close Proximity: With a fixed IoU threshold, it can incorrectly suppress valid detections for objects that are naturally close together or partially occluded (e.g., a crowd of people).
• Soft-NMS: Instead of outright removing overlapping boxes, it attenuates (lowers) their confidence scores based on the IoU with the higher-scoring box. This gives them a chance to be selected later if they are indeed a separate object.
• Weighted-NMS: Averages the coordinates of overlapping boxes (weighted by their confidence scores) to produce a refined, more accurate final bounding box, rather than just picking one.
• Adaptive-NMS: Dynamically adjusts the IoU threshold based on the local object density or detection score, becoming more permissive in crowded regions.

In real-time robotic perception, NMS is a lightweight but crucial component in the post-processing stage of an object detection pipeline (e.g., after a YOLO or SSD network). Its characteristics directly impact system performance:

• Deterministic & Fast: It adds minimal, predictable latency, which is essential for control loops.
• Directly Affects Metrics: The choice of IoU threshold influences the system's precision (fewer false positives) and recall (fewer missed objects).
• Downstream Impact: The cleaned bounding box list is passed to modules like Multi-Object Tracking (MOT), task planning, and visuomotor control. Erroneous suppression can cause an object's track to be lost, leading to planning failures.
• Embedded Optimization: For deployment on edge devices, NMS is often hand-optimized or integrated into inference engines like TensorRT to minimize CPU overhead.

Intersection over Union is the fundamental metric used by NMS to measure the overlap between predicted bounding boxes. It is calculated as the area of overlap divided by the area of union between two boxes.

• Role in NMS: NMS uses a predefined IoU threshold (e.g., 0.5) to determine if two boxes are detecting the same object. Boxes with an IoU above the threshold are considered duplicates.
• Impact on Performance: A higher IoU threshold results in fewer, more precise boxes but risks missing valid detections in crowded scenes. A lower threshold increases recall but can leave more duplicate predictions.

Object detection is the computer vision task of locating and classifying objects within an image, producing a set of bounding boxes and class labels. NMS is applied to the raw output of any object detector.

• Detector Types: NMS is used with both two-stage detectors (like Faster R-CNN) and single-stage detectors (like YOLO and SSD).
• Output Format: Detectors typically produce many overlapping candidate boxes per object, each with a confidence score. NMS's role is to prune this set to the single best box per object instance.
• Real-Time Constraint: For robotic perception, detectors must run at high frame rates, making the efficiency of the NMS post-processing step a critical component of the latency budget.

YOLO is a family of single-stage, real-time object detection models famous for their speed. NMS is an integral, non-negotiable component of the YOLO inference pipeline.

• Architectural Dependency: YOLO models predict a dense grid of bounding boxes. A single object can be detected by multiple grid cells, generating hundreds of overlapping predictions that must be consolidated by NMS.
• Evolution with NMS: Modern YOLO variants (v7, v8, v9) often integrate advanced NMS variants like Weighted NMS or Soft-NMS directly into their training and deployment frameworks to improve accuracy in occluded scenes.

Multi-Object Tracking is the task of locating multiple objects and maintaining their identities across video frames. NMS is used in the detection-to-track association step.

• Detection Filtering: Before tracks are updated or created, raw detections from each frame are passed through NMS to eliminate redundant boxes, providing a clean set of observations for the tracker.
• Interaction with Data Association: In tracking-by-detection paradigms, efficient NMS reduces computational load for subsequent steps like the Hungarian algorithm, which matches detections to existing tracks.

Soft-NMS is a popular refinement of the standard NMS algorithm that addresses its primary weakness: the hard suppression of overlapping high-confidence predictions.

• Mechanism: Instead of completely removing boxes with IoU above a threshold, Soft-NMS decays their confidence scores using a continuous, weight-decaying function (e.g., Gaussian).
• Advantage in Robotics: This is crucial for scenes with partial occlusion, where multiple valid but overlapping boxes may represent different parts of the same object or tightly grouped objects. It improves recall without introducing duplicate false positives.

The Hungarian algorithm is a combinatorial optimization algorithm that solves the assignment problem in polynomial time. In perception pipelines, it is often used after NMS for data association.

• Sequential Processing: NMS first cleans up the detections within a single frame. The Hungarian algorithm is then typically used to associate these cleaned detections with existing tracks across frames in a Multi-Object Tracking system.
• Cost Matrix: The algorithm minimizes a total cost, where cost is often based on metrics like IoU or Mahalanobis distance between a track's predicted state and a new detection. Efficient NMS upfront simplifies this cost matrix.