Black-Box Attack

The adversary interacts with the target model exclusively through an API or user interface, receiving either hard-label predictions (fraud/not fraud) or soft-label confidence scores. This mirrors real-world attack surfaces where proprietary fraud models are exposed only through public-facing endpoints. Attackers must infer the decision boundary by observing input-output relationships without any knowledge of architecture, parameters, or training data.

Black-box attacks bifurcate into two categories based on available output granularity:

• Score-based attacks: Exploit continuous confidence scores to estimate gradients numerically through finite-difference methods, enabling gradient-based optimization even without true gradients.
• Decision-based attacks: Operate solely on hard-label predictions, typically starting from a large perturbation and iteratively reducing it while maintaining misclassification. These are significantly more query-intensive but represent the hardest realistic threat model.

A powerful black-box strategy where the adversary trains a surrogate model on a locally curated dataset, generates adversarial examples against it using white-box methods, and then deploys them against the target model. This exploits the transferability property of adversarial examples—perturbations that fool one model often fool others trained on similar tasks. In financial fraud contexts, attackers may train surrogates on publicly available transaction datasets.

Real-world black-box attacks face practical limitations on the number of queries permitted before triggering rate limiting, anomaly detection, or monetary costs. Sophisticated attacks employ query-efficient strategies such as:

• Natural Evolution Strategies (NES): Estimate gradients using population-based sampling rather than coordinate-wise finite differences
• Bayesian optimization: Model the decision boundary with minimal queries
• Boundary attack: Walk along the decision boundary to find minimal perturbations without gradient estimation

In fraud detection contexts, black-box attacks manifest through:

• Transaction feature manipulation: Adversaries iteratively adjust transaction amounts, timing, or merchant categories to probe model thresholds
• Velocity pattern testing: Systematic querying to discover rate-limiting rules and temporal blind spots
• Geographic spoofing trials: Testing IP geolocation and shipping address correlations to map fraud rules These attacks exploit the online inference pipeline where each declined transaction provides a label.

Defending against black-box attacks requires monitoring query patterns for adversarial intent:

• Query anomaly detection: Flag unusual sequences of similar transactions with minor variations
• Randomized response smoothing: Add calibrated noise to confidence scores to obscure the true decision boundary
• Rate limiting and progressive delays: Throttle repeated queries from suspicious sources
• Honeypot models: Deploy decoy models with known vulnerabilities to detect probing behavior

Two subcategories of black-box attacks distinguished by API output access:

• Score-based attacks: Exploit continuous confidence scores or logits returned by the model. These enable gradient estimation via finite differences or zeroth-order optimization.
• Decision-based attacks: Operate with only the final hard-label prediction (e.g., 'fraud' or 'legitimate'). These are harder to execute, relying on boundary-tracing algorithms like HopSkipJumpAttack that walk the decision boundary with minimal queries. Financial systems that hide scores still face decision-based threats.

A core metric in black-box attack evaluation measuring how many API calls are required to generate a successful adversarial example. Query-limited settings are the most realistic threat model for financial systems, where rate limiting and per-query costs constrain attackers. State-of-the-art attacks like Square Attack achieve high success rates with hundreds rather than thousands of queries. Defenders monitor query patterns—bursts of similar transactions—as a detection signal for ongoing black-box attacks.

A defensive mechanism designed to distinguish between clean inputs and adversarial examples before they reach the main fraud model. Detection strategies include:

• Feature squeezing: Reducing input dimensionality to collapse adversarial perturbations
• Statistical tests: Comparing input distributions against training data baselines
• Auxiliary classifiers: Training a separate binary detector on adversarial vs. clean samples Black-box attackers must now evade both the detector and the primary model, increasing attack complexity and query cost.