Label-Only Inversion

Unlike standard model inversion which exploits confidence scores, label-only attacks rely solely on the hard-label prediction (e.g., 'dog' vs. 'cat'). The attacker iteratively perturbs a random input and observes whether the predicted class flips. By probing the decision boundary from multiple directions, the adversary can reconstruct a prototypical image that sits precisely at the classification threshold, revealing the model's internal representation of that class without ever seeing a probability distribution.

Label-only attacks are inherently query-intensive because each iteration provides only a single bit of information (correct/incorrect class). Attackers optimize for query efficiency by:

• Using boundary-proximity algorithms that estimate gradient direction from label changes
• Employing evolutionary strategies to guide the search
• Leveraging natural image priors (e.g., total variation, color consistency) to constrain the search space A successful attack may require thousands to millions of queries, making query auditing a viable defense.

The primary defense that forces attackers into a label-only regime is confidence score masking, where the model API returns only the top-k class labels without probability vectors. This defense truncates the rich signal available to standard inversion attacks. However, label-only attacks demonstrate that masking alone is insufficient—determined adversaries can still reconstruct recognizable class prototypes. Effective defense requires combining masking with differential privacy, output perturbation, or query rate limiting.

Label-only inversion produces class-level representatives, not specific training examples. The reconstructed images capture common features the model associates with a class—such as general shapes, textures, and color distributions—but lack instance-specific details like faces, license plates, or background elements. This distinguishes label-only attacks from gradient inversion or Deep Leakage from Gradients (DLG), which can recover pixel-accurate training samples. The privacy risk is class-level leakage rather than individual record exposure.

Label-only inversion shares algorithmic foundations with decision-based adversarial attacks (e.g., Boundary Attack, HopSkipJump). Both operate in the hard-label black-box setting and iteratively walk along the decision boundary. The key difference is the objective: adversarial attacks seek minimally perturbed misclassified inputs, while label-only inversion seeks maximally prototypical class representations. Techniques from one domain—such as gradient estimation via finite differences and boundary-following algorithms—directly transfer to the other.

Training models with an information bottleneck objective compresses input data into a latent representation that is maximally informative for the task while minimizing mutual information with the original input. This naturally limits the fidelity of label-only reconstructions because the model discards fine-grained details unnecessary for classification. Related techniques include:

• Maximal Coding Rate Reduction (MCR2) for discriminative compressed features
• Variational autoencoder preprocessing to strip sensitive high-frequency signals
• Dropout during training to introduce representational uncertainty

The broader class of attacks that reconstruct representative samples of a target class or specific training data points by exploiting access to a trained model's parameters and confidence scores. Label-only inversion is a constrained variant of this attack that operates without access to confidence scores, making it harder to detect and defend against. The attacker iteratively queries the model and uses only the predicted class label to guide the reconstruction process.

A defense mechanism that limits the information leaked by a model by truncating or rounding the prediction vector to return only the top-k class labels instead of full probability distributions. While effective against standard model inversion attacks that rely on confidence scores, label-only inversion attacks are specifically designed to bypass this defense by operating solely on hard-label predictions.

An attack that determines whether a specific data record was part of a model's training set by analyzing the model's prediction behavior on that record. Label-only inversion shares a similar threat model with MIA, as both exploit the model's output behavior. Defenses against MIA, such as differential privacy and output perturbation, can also degrade the fidelity of label-only inversion reconstructions.

A mathematical framework that provides provable privacy guarantees by injecting calibrated noise into the computation, ensuring the output distribution is nearly identical whether or not a single individual's data is included. DP-SGD is the primary training-time defense against label-only inversion, as it bounds the influence of individual training examples by clipping per-sample gradients and adding Gaussian noise, directly limiting what can be reconstructed.

A runtime defense that monitors incoming API queries to a model, detecting and blocking anomalous sequences or high-frequency requests characteristic of model inversion or extraction attacks. Since label-only inversion requires a large number of queries to iteratively refine the reconstructed image, query auditing can detect and throttle these attack patterns by enforcing rate limits and analyzing query distribution entropy.

A training methodology that augments the loss function with a penalty term designed to minimize the success of an adversary attempting to reconstruct inputs or infer membership. Techniques like Mixup training and defensive distillation smooth decision boundaries and reduce the information content of model outputs, making it harder for label-only inversion attacks to extract high-fidelity reconstructions from hard-label predictions.