DETR (DEtection TRansformer)

Unlike traditional models that generate a dense grid of proposals, DETR predicts a fixed-size set of N bounding boxes and class labels in a single pass. The model is forced to reason globally about the entire image context and the relationships between objects, rather than making local, overlapping guesses. This eliminates duplicate detections by design, making the post-processing step of Non-Maximum Suppression (NMS) entirely obsolete.

The core training mechanism is the Hungarian algorithm, which finds a one-to-one optimal assignment between the N predicted objects and the ground truth annotations. The loss function is computed only on these matched pairs, penalizing both class misclassifications and bounding box inaccuracies. Unmatched predictions are correctly penalized as 'no object' (∅). This set-based global loss ensures that each target object has exactly one predictor, enforcing unique detections.

DETR uses a standard Transformer architecture. A CNN backbone first extracts a feature map, which is flattened and supplemented with positional encodings. The transformer encoder then processes these features with global self-attention, allowing each pixel to contextualize itself with the entire image. The decoder takes a set of learned object queries (positional embeddings) and cross-attends to the encoder output, directly generating the final predictions in parallel.

The decoder receives N learned positional embeddings called object queries. These are not tied to specific spatial locations but are learned vectors that, through cross-attention, specialize in predicting objects in different regions and at different scales. Crucially, all N predictions are generated simultaneously in parallel, rather than sequentially. This is a fundamental departure from autoregressive models and enables a truly holistic scene understanding.

DETR removes the need for many manually designed components that encode human knowledge:

• No Anchor Boxes: It predicts bounding boxes directly without reference to pre-defined templates of specific scales or aspect ratios.
• No Region Proposal Network (RPN): The architecture does not require a separate network to suggest candidate object locations.
• No NMS: The bipartite matching loss enforces unique predictions, eliminating the heuristic post-processing step entirely.

The primary limitation of the original DETR is its slow training convergence, particularly for small objects. Standard transformers struggle to focus on fine-grained details in the early stages of training. This issue was later addressed by Deformable DETR, which replaces standard attention with deformable attention modules that attend only to a sparse set of learned sampling points around a reference, drastically accelerating convergence and improving performance on small object detection.

The set-based global loss that replaces Non-Maximum Suppression (NMS). DETR uses the Hungarian algorithm to find a one-to-one optimal assignment between predicted and ground truth objects before calculating loss. This forces the model to predict exactly one box per object, eliminating duplicate detections natively. The loss combines a classification term for label matching and a box loss (L1 + Generalized IoU) for spatial alignment.

A fixed set of learned positional embeddings that serve as the decoder's input. Each query specializes in predicting one object by attending to different spatial regions of the image. Unlike anchor boxes, these queries are content-agnostic initially but learn to focus on specific object sizes, locations, and class co-occurrence patterns during training. The number of queries (typically 100) defines the maximum objects DETR can detect in a single pass.

DETR uses a standard Transformer architecture adapted for vision. The encoder processes a flattened CNN feature map with positional encodings, allowing global self-attention across the entire image. The decoder takes object queries and cross-attends to encoder outputs, simultaneously predicting all object classes and bounding boxes in parallel. This global context enables modeling of co-occurrence relationships (e.g., a nodule near a rib) that anchor-based methods miss.

Since transformers are permutation-invariant, DETR adds fixed sinusoidal positional encodings to the flattened image features before the encoder. These encodings inject spatial awareness by representing each pixel's (x, y) coordinates as a unique frequency pattern. This allows the self-attention mechanism to reason about relative spatial relationships—critical for understanding that two features belong to the same anatomical structure.

An evolution of DETR that replaces dense global attention with deformable attention modules. Instead of attending to all spatial locations, each query attends only to a small set of learned sampling points around a reference point. This dramatically reduces computational complexity from O(n²) to O(n), accelerates convergence by 10x, and improves small object detection—critical for identifying micro-calcifications and early-stage lesions in mammography.

DETR reframes object detection as a direct set prediction problem rather than a sequential proposal-and-refine pipeline. The model outputs a set of (class, box) tuples in a single forward pass without any post-processing. This end-to-end philosophy eliminates hand-crafted components like anchor generation, Region Proposal Networks, and Non-Maximum Suppression, making the architecture simpler and more maintainable for clinical deployment pipelines.