Deep Radiomics

Unlike traditional radiomics which requires explicit definition of texture matrices (GLCM, GLRLM), deep radiomics uses convolutional neural networks to learn a hierarchy of features directly from raw voxel data. Lower layers detect edges and corners, middle layers aggregate textures, and deep layers encode high-level semantic abstractions such as tumor heterogeneity patterns. This bypasses the intensity discretization and feature harmonization bottlenecks that plague engineered pipelines.

A dominant paradigm in deep radiomics involves fine-tuning models pre-trained on massive natural image datasets like ImageNet. The early convolutional filters learned on photographs—edge detectors, color blobs, Gabor-like filters—transfer surprisingly well to medical imaging modalities. This parameter-efficient fine-tuning strategy mitigates the chronic data scarcity in medical imaging, where annotated cohorts rarely exceed a few hundred patients. Common architectures include ResNet-50, DenseNet-121, and EfficientNet backbones.

To overcome the limitation of transfer learning from non-medical domains, self-supervised learning formulates pretext tasks directly on unlabeled medical images. Examples include:

• Context restoration: predicting missing patches
• Rubik's cube recovery: unscrambling shuffled image tiles
• Contrastive learning: maximizing agreement between augmented views of the same scan These methods learn domain-specific anatomical and textural representations without requiring radiologist annotations, producing feature extractors that outperform ImageNet transfer for downstream tasks.

Deep radiomics architectures natively support the fusion of heterogeneous data streams. Multi-scale fusion integrates features extracted at different receptive field sizes, capturing both fine-grained cellular patterns and macroscopic tumor morphology simultaneously. Multi-modal fusion combines imaging data with genomic profiles, pathology slides, or clinical variables through late-fusion layers or cross-attention mechanisms. This enables the construction of composite biomarkers that correlate imaging phenotypes with molecular subtypes.

A critical challenge for clinical adoption is the 'black box' nature of deep features. Modern deep radiomics pipelines integrate class activation mapping (CAM) and Grad-CAM to generate heatmaps localizing which anatomical regions drove a prediction. Attention visualization in transformer-based architectures reveals which spatial relationships the model deemed salient. These techniques provide the algorithmic explainability required for regulatory submissions and build clinician trust by mapping abstract deep features back to interpretable anatomical correlates.

State-of-the-art deep radiomics often employs hybrid architectures that concatenate handcrafted IBSI-compliant features with deep learned representations before final classification. The engineered features provide explicit physical meaning (e.g., GLCM homogeneity quantifies local uniformity), while the deep features capture latent textural phenotypes not expressible by mathematical formulas. This dual-pathway approach consistently outperforms either method alone in tasks like overall survival prediction and pathological complete response assessment.

The foundational architecture of deep radiomics. CNNs apply hierarchical filters that automatically learn to detect edges in early layers, textures in middle layers, and complex anatomical structures in deep layers. Unlike handcrafted radiomics, CNNs learn translationally invariant features directly from raw voxel data, eliminating the need for explicit segmentation or intensity discretization in the feature extraction step. Common architectures include ResNet for skip connections that preserve spatial detail and DenseNet for feature reuse across layers.

A critical strategy that addresses the scarcity of labeled medical datasets. Models are pre-trained on large natural image datasets like ImageNet, then fine-tuned on medical imaging data. The early convolutional layers that detect universal features (edges, curves) transfer effectively, while deeper layers adapt to domain-specific patterns like tumor heterogeneity and tissue interfaces. This approach reduces the required training data by orders of magnitude and significantly accelerates convergence compared to training from random initialization.

The defining characteristic that distinguishes deep radiomics from traditional radiomics. Rather than computing predefined mathematical descriptors (GLCM, GLRLM), the network learns task-optimized feature representations through backpropagation. Key advantages include:

• No feature engineering bottleneck: The model discovers relevant patterns autonomously
• Joint optimization: Feature extraction and classification are trained simultaneously
• Higher discriminative power: Learned features often outperform engineered features on complex prognostic tasks

A fundamental design choice in deep radiomics. 3D CNNs process entire volumetric data (e.g., CT/MRI volumes) using 3D convolutional kernels, capturing full spatial context and inter-slice relationships. 2.5D approaches use three orthogonal 2D CNNs (axial, sagittal, coronal) and fuse their predictions, offering a compromise between computational efficiency and volumetric awareness. 3D models provide superior context for irregular tumor boundaries but require significantly more GPU memory and training data.

Architectural components that allow deep radiomics models to focus on diagnostically relevant regions while suppressing irrelevant anatomy. Spatial attention learns to weight different voxel locations based on their importance to the task. Channel attention (e.g., Squeeze-and-Excitation blocks) recalibrates feature maps to emphasize informative feature channels. In oncology applications, attention maps can serve as interpretability tools, highlighting tumor subregions that drive the model's prediction.

An emerging paradigm that learns representations from unlabeled medical images before fine-tuning on limited labeled data. Common pretext tasks include:

• Context restoration: Predicting missing image patches
• Contrastive learning: Maximizing agreement between augmented views of the same scan
• Rotation prediction: Classifying the applied rotation angle This approach exploits the vast repositories of archived clinical imaging data, reducing dependence on expensive expert annotations while learning domain-specific anatomical priors.