Gap Attack

The attack operates on the principle that models exhibit a wider confidence gap between the predicted class and the next highest confidence for training data compared to non-training data. This metric, often called the prediction gap or margin, serves as the primary signal for membership inference. An adversary computes this gap for each query and sets a threshold to classify records as members or non-members.

Gap attacks are highly sensitive to a model's calibration state. A well-calibrated model produces confidence scores that reflect true empirical likelihoods, potentially narrowing the exploitable gap. Conversely, overconfident mispredictions on out-of-distribution data create artificially large margins that leak membership information. Modern variants normalize the gap using reference distributions from shadow models to improve attack accuracy against calibrated targets.

This attack requires only hard-label or top-K prediction access to the target model, making it a highly practical black-box threat. The adversary does not need gradients, logits, or full confidence vectors—only the predicted class and the next most likely class. This minimal data requirement allows the attack to bypass many common API defenses that restrict output detail, such as confidence masking or top-1-only responses.

Attack performance hinges on selecting an optimal gap threshold to separate members from non-members. The adversary typically trains shadow models on datasets drawn from the same distribution as the target's training data to empirically derive this threshold. Advanced implementations use per-class thresholds or adaptive thresholding based on input difficulty, recognizing that the gap distribution varies significantly across different data subpopulations.

Mitigations target the information leakage in prediction margins. Differential privacy during training (DP-SGD) limits memorization, reducing the gap disparity. Output perturbation adds calibrated noise to confidence scores to obscure the margin signal. Adversarial regularization techniques explicitly penalize large confidence gaps on non-training data during model optimization, directly hardening the model against this specific attack vector.

The attack's efficacy is a direct function of model overfitting. Models that memorize training examples exhibit systematically larger prediction gaps on those examples compared to test data. The gap attack essentially operationalizes overfitting detection as a privacy vulnerability. Regularization techniques like weight decay, dropout, and early stopping that reduce overfitting inherently diminish the attack's success rate by compressing the margin distribution between training and test sets.