Sharpness-Aware Minimization (SAM)

SAM reformulates the standard empirical risk minimization objective into a minimax problem. Instead of simply finding parameters w with low loss, SAM seeks parameters where the loss remains low even when weights are perturbed within a defined neighborhood.

• The objective is: min_w max_{||ε|| ≤ ρ} L(w + ε)
• The inner maximization identifies the worst-case weight perturbation ε within a Euclidean ball of radius ρ.
• The outer minimization then updates weights to be robust against this adversarial perturbation.
• This explicitly penalizes sharp, narrow minima where small weight changes cause large loss spikes.

SAM computes a two-step gradient that decouples the direction of perturbation from the direction of weight update. This is computationally efficient, requiring only two forward-backward passes per iteration.

• Step 1: Compute the gradient at the current weights and ascend to the worst-case perturbation: ε = ρ * ∇L(w) / ||∇L(w)||.
• Step 2: Compute the gradient at the perturbed weights w + ε and apply this as the actual descent direction.
• The resulting update implicitly minimizes both the loss value and the local curvature.
• This doubles the computational cost per step compared to standard SGD but converges to flatter regions.

Flat minima discovered by SAM provide inherent resilience to post-training quantization (PTQ) . When weights are rounded to low-precision integers, the loss perturbation is bounded.

• In a sharp minimum, a small weight perturbation from quantization can cause a catastrophic loss increase.
• In a flat minimum, the same perturbation magnitude keeps the loss within an acceptable basin.
• Empirically, SAM-trained models suffer 50-75% less accuracy degradation after INT8 quantization compared to standard SGD-trained equivalents.
• This property is critical for deploying neural receivers on integer-only edge hardware without quantization-aware retraining.

The original SAM formulation can be sensitive to the choice of batch size. m-Sharpness extends the concept by measuring sharpness over multiple independent minibatches, decoupling the perturbation radius from the batch statistics.

• Standard SAM sharpness is computed on a single minibatch, which can be noisy.
• m-Sharpness averages the worst-case loss over m independent data shards.
• This provides a more stable estimate of the true population sharpness.
• Models optimized for m-Sharpness exhibit superior generalization on unseen test distributions, a critical requirement for RF signals in dynamic electromagnetic environments.

Several adaptive variants address the sensitivity of SAM to the perturbation radius hyperparameter ρ . These methods dynamically scale the perturbation per-parameter based on weight magnitude or gradient statistics.

• ASAM (Adaptive SAM): Scales the perturbation neighborhood by the weight norm, ensuring the perturbation is proportional to parameter scale.
• Fisher SAM: Uses the Fisher information matrix to define an ellipsoidal rather than spherical perturbation region.
• GSAM (Gradient SAM): Decomposes the update into a sharpness-reducing component and a loss-minimizing component, optimizing both simultaneously.
• These variants reduce the need for extensive hyperparameter tuning while maintaining flat-minima convergence.

SAM is particularly valuable for neural receiver architectures deployed on resource-constrained edge hardware. The flat minima property directly addresses the quantization challenges of complex-valued IQ data.

• Automatic Modulation Classification: SAM-trained classifiers maintain >95% accuracy after 4-bit weight quantization, compared to <80% for standard training.
• Channel Estimation: Flat-minima neural estimators preserve phase coherence after compression, critical for coherent demodulation.
• RF Fingerprinting: SAM improves the stability of emitter identification features under hardware-aware pruning and INT8 conversion.
• The technique integrates seamlessly with standard training pipelines for transformer-based and convolutional signal processors.

A training paradigm that simulates low-precision arithmetic during the forward pass, allowing the model to adapt its weight distribution to minimize quantization error. While SAM finds flat minima that are structurally tolerant of perturbation, QAT actively teaches the model to expect INT8 rounding noise. When combined, SAM provides a superior initialization landscape for QAT, often yielding higher post-quantization accuracy than QAT alone on a sharp minimum.

A compression technique that converts a pre-trained 32-bit floating-point model to 8-bit integers without retraining, using a small calibration dataset to set clipping ranges. Models trained with SAM exhibit significantly less degradation under PTQ because their parameters reside in wide basins where small perturbations—like rounding to INT8—do not cause the loss to spike. This makes SAM a critical pre-training step for teams that cannot afford QAT's retraining cost.

The systematic removal of low-magnitude connections to reduce model size and compute. SAM's flat minima directly benefit pruning in two ways:

• Higher sparsity tolerance: Flat regions allow more weights to be zeroed without crossing a loss barrier
• Improved fine-tuning stability: The remaining weights sit in a well-conditioned basin, enabling faster recovery with minimal retraining This synergy is especially critical for unstructured sparsity on RF models where phase coherence must be preserved.

A compression method where a compact student model is trained to replicate the soft output distribution of a larger teacher model. When the teacher is trained with SAM, it produces smoother, more calibrated soft labels that encode dark knowledge about class boundaries. This richer supervisory signal allows the student to achieve higher accuracy than when distilled from a teacher that converged to a sharp, overconfident minimum.

A gradient approximation technique that passes the gradient through non-differentiable quantization nodes unchanged during backpropagation. SAM and STE interact at a fundamental level: SAM's inner perturbation step requires computing gradients at displaced points, and the STE ensures these gradients flow cleanly through simulated quantization operators. This makes SAM compatible with quantization-aware training pipelines that rely on STE for end-to-end learning.

A diagnostic technique that plots the loss function along random directions in weight space to reveal the geometry of convergence. SAM-trained models exhibit characteristically flat basins when visualized—the loss remains low across a wide neighborhood rather than spiking immediately. Tools like filter normalization plots are used to empirically verify that SAM has successfully located a flat minimum before committing to expensive compression and deployment workflows.