Black-Box Attack

The adversary relies exclusively on the input-output mapping of the target API. Since gradient information is unavailable, attackers use zeroth-order optimization methods to estimate gradients through finite differences or random search. Techniques like Natural Evolution Strategies (NES) sample perturbations around a current point to approximate the gradient direction, enabling the construction of adversarial examples without ever touching the model's internal weights.

Black-box attacks are categorized by the richness of the model's output:

• Score-based: The API returns continuous confidence scores or logits. The attacker uses these precise probabilities to guide the search for adversarial examples.
• Decision-based: The API returns only the final class label (hard-label). This is a significantly harder setting, often requiring random-walk algorithms like the Boundary Attack that start from a large perturbation and iteratively reduce it while staying adversarial.

A critical vulnerability exploited in black-box settings is adversarial transferability. An attacker trains a local substitute model on a synthetic dataset labeled by repeatedly querying the target API. Adversarial examples crafted against this white-box substitute often transfer and fool the original black-box model. This is because both models learn similar decision boundaries, even with different architectures.

Real-world black-box attacks must balance effectiveness against query budgets. Defensive mechanisms like rate limiting, IP throttling, and per-query costs directly constrain the attacker. Efficient attacks like Square Attack use randomized search heuristics that achieve high success rates with orders of magnitude fewer queries than earlier coordinate-descent methods. The query efficiency metric (queries per successful attack) is the primary benchmark.

Beyond misclassification, black-box access enables functionality extraction. An adversary can train a student model to mimic the target's decision boundary by using the API as a labeling oracle. With sufficient queries, the attacker reconstructs a high-fidelity clone that not only performs equivalently on the original task but also serves as a white-box surrogate for crafting future adversarial inputs or extracting proprietary intellectual property.

In the strictest decision-based setting, the Boundary Attack algorithm operates by initializing with a sample already classified as the target adversarial class. It then performs a random walk along the decision boundary, iteratively reducing the perturbation magnitude while maintaining the adversarial classification. The attack requires no gradient information and works against any model that exposes a label, making it a universal black-box threat.

Attackers systematically query commercial ML APIs (e.g., image classification, NLP) to build a functionally equivalent surrogate model. By sending carefully crafted inputs and recording predictions, adversaries steal proprietary model intellectual property. Techniques include:

• Equation-solving attacks that recover parameters layer by layer
• Active learning strategies to minimize query budgets
• Decision boundary reconstruction via adaptive sampling

Real-world impact: A stolen model enables white-box attacks, evasion, and eliminates per-query API revenue. Researchers demonstrated extracting a complete model from BigML and Amazon ML platforms with fewer than 6,500 queries.

Adversaries determine whether a specific individual's data was used to train a model by analyzing prediction confidence vectors. Overfitted models output higher confidence on training samples. Attack methodology:

• Train shadow models on similar data distributions to mimic target behavior
• Build a binary classifier on shadow model outputs to distinguish members from non-members
• Apply the attack classifier to target model outputs

Privacy implications: Confirmed membership in sensitive datasets (medical records, financial transactions) constitutes a direct privacy violation. Landmark research demonstrated 74% accuracy inferring membership in CIFAR-100 training data using only black-box access.

When models return only hard labels (class predictions) instead of confidence scores, attackers exploit decision boundaries through iterative perturbation. Key techniques:

• Boundary attack: Start with an adversarial sample and walk along the decision boundary toward the target
• HopSkipJump attack: Estimate gradient direction using only binary decisions
• Spatial transformations: Apply rotation, translation, or elastic deformations imperceptible to humans

Practical demonstration: Researchers bypassed commercial facial recognition systems by applying subtle image warping, causing misclassification without pixel-level noise. These attacks require no gradient access and transfer across architectures.

Large language models memorize and can regurgitate training data when prompted adversarially. Extraction strategies include:

• Divergence attacks: Prompt the model to repeat rare sequences until memorized content emerges
• Prefix matching: Supply partial training text and observe completions
• Membership testing via perplexity: Measure output likelihood to infer training inclusion

Documented cases: Researchers extracted verbatim PII including names, phone numbers, and addresses from GPT-2 by prompting with known prefixes. The attack required only API access and recovered 600+ unique memorized sequences from a single model.

Attackers deploy physical-world adversarial objects that cause consistent misclassification across viewpoints and lighting conditions. Unlike digital perturbations, these patches are printed and placed in the environment. Notable examples:

• Stop sign stickers causing autonomous vehicle classifiers to read "Speed Limit 45"
• Eyeglass frames defeating facial recognition systems
• Infrared LED arrays invisible to humans but blinding to surveillance cameras

Operational reality: These attacks succeed because black-box models learn brittle features correlated with, but not causally related to, object identity. Patches exploit this by presenting high-activation patterns that dominate the classifier's attention mechanism.

Attackers reconstruct representative training samples using only prediction scores through gradient estimation and optimization. Methodology:

• Estimate the gradient of the model's confidence with respect to input using finite differences or evolutionary strategies
• Optimize a randomly initialized image to maximize class confidence
• Apply natural image priors to produce realistic reconstructions

Privacy breach severity: Researchers reconstructed recognizable faces from facial recognition models trained on AT&T and Yale datasets using fewer than 8,000 queries per image. The reconstructed faces were visually matched to training subjects by human evaluators.

An adversarial attack executed with full knowledge of the target model's internal architecture, gradients, and trained parameters. The attacker can compute exact loss gradients to craft highly optimized perturbations.

• Access Level: Model internals, weights, training algorithm
• Efficiency: Highest attack success rate due to gradient access
• Contrast: Directly opposite to black-box attacks in the threat model spectrum

An attack that determines whether a specific data record was part of the model's training dataset by analyzing prediction confidence scores and output distributions. This is a common black-box attack vector since it requires only query access.

• Signal: Overfitted models leak higher confidence on training members
• Technique: Shadow model training to calibrate inference classifier
• Privacy Impact: Directly violates data membership confidentiality

The property where adversarial examples crafted against one surrogate model also fool a different target model. This enables black-box attacks by training a local substitute and transferring the generated perturbations.

• Mechanism: Shared decision boundaries across similar architectures
• Enhancement: Ensemble-based attacks improve transfer success rates
• Implication: Defenses must account for cross-model vulnerability

A black-box attack technique that estimates gradients through finite differences by observing output changes from carefully crafted input perturbations. Methods include zeroth-order optimization and natural evolution strategies.

• Score-Based: Requires confidence scores, not just hard labels
• Decision-Based: Operates with only final class predictions
• Cost: High query budgets needed for accurate gradient approximation

An attack exploiting physical information leakage—timing, power consumption, electromagnetic emissions, or memory access patterns—to extract model secrets without direct query access. This represents a distinct threat vector outside the query-based black-box paradigm.

• Channels: Cache timing, power traces, acoustic emanations
• Target: Model architecture, weights, or hyperparameters
• Defense: Constant-time operations and hardware isolation