Sharpness-Aware Minimization (SAM)

SAM reformulates the standard optimization objective by seeking parameters that minimize not just the training loss, but the maximum loss within a defined neighborhood. This is achieved by solving a minimax optimization problem: an inner maximization step identifies the worst-case perturbation in parameter space, and an outer minimization step updates the weights to reduce the loss at that perturbed location. This dual objective explicitly penalizes sharp minima—regions where the loss landscape exhibits high curvature—and biases convergence toward flat minima that are less sensitive to small parameter variations.

The hyperparameter ρ (rho) defines the radius of the neighborhood over which the inner maximization is performed. It controls the trade-off between sharpness penalization and convergence speed:

• Small ρ: Behaves similarly to standard SGD, converging to sharper minima with potentially higher training accuracy but lower robustness.
• Large ρ: Aggressively seeks flatter regions, improving generalization and adversarial robustness but may slow convergence or underfit.
• Typical values: Range from 0.01 to 0.2 for normalized inputs; optimal values are dataset and architecture dependent. The perturbation is typically scaled by the parameter norm to maintain scale-invariance across layers.

Flat minima discovered by SAM exhibit a direct correlation with adversarial robustness in modulation classification tasks. The underlying principle is that a classifier operating in a flat loss basin requires a larger input perturbation to cross a decision boundary compared to one in a sharp basin. Key mechanisms:

• Gradient obfuscation resistance: Unlike some adversarial training methods, SAM does not rely on gradient masking, making it harder to circumvent with adaptive attacks.
• Complementary defense: SAM can be combined with adversarial training (e.g., PGD-based) to achieve robustness against both white-box and black-box evasion attacks.
• Empirical evidence: Studies show SAM-trained models maintain higher classification accuracy under FGSM and PGD attacks compared to standard SGD-trained baselines.

The SAM algorithm proceeds in two distinct steps per iteration:

1. First forward/backward pass: Compute the gradient of the loss at the current parameters w. Scale this gradient to obtain the perturbation ε̂(w) that maximizes loss within the ρ-neighborhood.
2. Second forward/backward pass: Evaluate the loss gradient at the perturbed parameters w + ε̂(w). Use this gradient to update the original parameters w. This effectively doubles the computational cost per iteration compared to standard SGD. Variants like ASAM (Adaptive SAM) introduce normalization to make the sharpness definition scale-invariant, improving performance on architectures with varying weight scales.

While SAM is prominently studied for adversarial robustness, its primary contribution is to standard generalization—improving test accuracy on clean, unperturbed data. In modulation classification, this translates to:

• Improved performance under channel impairments: Models generalize better to unseen fading, noise, and frequency offset conditions without explicit augmentation.
• Reduced overfitting on small RF datasets: SAM acts as a strong implicit regularizer, particularly valuable when labeled signal data is scarce.
• State-of-the-art results: SAM and its variants have achieved top performance on benchmarks including CIFAR-10, CIFAR-100, and ImageNet, often surpassing sophisticated augmentation strategies alone.

SAM can be integrated into existing adversarial training workflows to produce models that are robust to both natural distribution shifts and deliberate adversarial perturbations. A combined approach typically:

• Uses SAM as the outer optimizer to find flat minima.
• Generates adversarial examples (via PGD or FGSM) on-the-fly during training.
• Updates parameters using the adversarial loss gradient at the SAM-perturbed location. This dual strategy addresses two distinct robustness concerns simultaneously: parameter-space sharpness (handled by SAM) and input-space perturbation sensitivity (handled by adversarial training). The resulting models demonstrate superior certified robustness under randomized smoothing frameworks.