Black-box Attack

The attacker has no access to the model's architecture, weights, gradients, or training data. All information must be gleaned through query access alone, making this the most realistic attack scenario for publicly exposed APIs and commercial AI services. This constraint forces attackers to treat the target as a complete oracle.

Attackers systematically probe the model by submitting carefully crafted inputs and observing outputs. Key strategies include:

• Score-based attacks: Exploiting confidence scores or logit outputs
• Decision-based attacks: Working only with hard-label predictions
• Transfer-based attacks: Using a surrogate model trained on query responses to generate adversarial examples

A common black-box strategy involves building a substitute model that mimics the target's decision boundary. The attacker:

1. Queries the target model with synthetic inputs
2. Labels them with the target's predictions
3. Trains a local surrogate on this labeled dataset
4. Generates adversarial examples against the surrogate, exploiting transferability to fool the original target

Black-box attacks bifurcate based on output granularity:

Score-based: The API returns probability vectors or confidence scores, enabling gradient estimation via finite differences or zeroth-order optimization.

Decision-based: Only the final class label is returned, requiring more sophisticated techniques like boundary attacks that walk along the decision boundary to find minimal perturbations.

Real-world black-box attacks face practical limitations:

• Query budgets: APIs often impose rate limits or per-query costs
• Detection risk: High query volumes trigger anomaly detection systems
• Stateful defenses: Systems that track and block suspicious query patterns

Efficient attacks like Square Attack and HopSkipJump minimize queries while maintaining high success rates.

Black-box attacks manifest in production systems through:

• Cloud API exploitation: Attacking GPT-4, Claude, or hosted vision APIs
• Physical-world attacks: Printing adversarial patches tested against deployed surveillance or autonomous vehicle systems
• NLP model attacks: Using synonym substitution frameworks like TextFooler against sentiment analysis or content moderation endpoints
• Audio adversarial examples: Crafting perturbations that fool voice authentication systems

The fundamental distinction in adversarial security is the attacker's level of access. A white-box attack assumes full knowledge of the model's architecture, parameters, and gradients, enabling powerful gradient-based methods like PGD. A black-box attack operates with zero internal knowledge, relying solely on querying the model to observe input-output pairs. This constraint forces attackers to use query-based or transfer-based strategies, making black-box attacks more realistic for external threats but often less sample-efficient.

Without access to gradients, black-box attackers must estimate them through systematic probing. Key strategies include:

• Score-based attacks: Exploit confidence scores to approximate gradients via finite differences.
• Decision-based attacks: Use only the hard-label prediction to walk the decision boundary, often via iterative refinement.
• Transfer attacks: Generate adversarial examples on a surrogate white-box model and deploy them against the target, exploiting the transferability property.
• Search-based attacks: Employ genetic algorithms or random search to find perturbations that flip labels.

Defending against black-box attacks requires strategies that do not rely on obscuring gradients. Core defenses include:

• Adversarial Training: Augmenting training data with adversarial examples to harden the decision boundary.
• Randomized Smoothing: Adding Gaussian noise during inference to create a provably robust, smoothed classifier with certified guarantees.
• Defensive Distillation: Training on soft labels to smooth the loss landscape and reduce gradient estimation accuracy.
• Gradient Masking: Techniques that obscure or destroy gradient information, though many are vulnerable to circumvention by black-box methods.

Quantifying black-box attack effectiveness and model resilience requires standardized metrics:

• Attack Success Rate (ASR): The percentage of adversarial examples that cause misclassification.
• Query Efficiency: The average number of model queries required to generate a successful adversarial example. Lower is more dangerous.
• Perturbation Magnitude: Measured via Lp-norms (L2, L∞) to quantify the perceptual distortion applied to the input.
• Robust Accuracy: The model's classification accuracy on an adversarially perturbed test set, the primary measure of defensive efficacy.

In NLP, black-box constraints are the default for external attackers facing API-gated LLMs. Key attack vectors include:

• TextFooler: Identifies semantically important words and replaces them with synonyms to flip classification without changing meaning.
• Prompt Injection: Crafting malicious inputs that override system instructions, exploiting the model's instruction-following behavior.
• Jailbreaking: Iteratively refining prompts to bypass safety alignment and content filters, often using role-playing or encoding tricks.
• Perplexity Filters: A defense that detects anomalous, high-perplexity adversarial suffixes characteristic of automated attacks like GCG.

The adversarial ML community relies on standardized libraries for reproducible research:

• Adversarial Robustness Toolbox (ART): A Linux Foundation project providing unified implementations of attacks, defenses, and certifications across modalities.
• CleverHans: A widely-adopted benchmarking library with standardized attack and defense reference implementations.
• MITRE ATLAS: A knowledge base of adversarial tactics, techniques, and case studies modeled after MITRE ATT&CK, providing a shared taxonomy for AI security incidents.
• Foolbox: A Python toolbox for creating adversarial examples with a focus on black-box decision-based attacks.