Threat Model

Defines the security violation the attacker aims to achieve. This dimension specifies whether the adversary seeks integrity violation (triggering misclassification without detection), availability violation (causing denial of service), or privacy violation (extracting sensitive training data).

• Targeted: Force a specific incorrect output (e.g., classify a stop sign as a speed limit sign)
• Untargeted: Cause any misclassification
• Model Extraction: Steal the model's functionality via API queries
• Membership Inference: Determine if a record was in the training set

Characterizes the information available to the attacker about the target system. This spectrum ranges from white-box (full access to architecture, weights, and gradients) to black-box (only input-output query access).

• White-Box: Attacker has complete knowledge of model architecture, parameters, and training algorithm. Enables gradient-based attacks like FGSM and PGD.
• Black-Box: Attacker can only query the model API and observe outputs. Requires transfer attacks or score-based optimization.
• Gray-Box: Partial knowledge, such as knowing the architecture but not the weights, or having access to confidence scores but not gradients.

Specifies the constraints and resources available to the attacker when manipulating inputs. This is formalized mathematically using Lp norm bounds that limit perturbation magnitude.

• L-infinity (L∞): Limits the maximum change to any single pixel or feature. Common bound: ε = 8/255 for images.
• L2: Constrains the Euclidean distance of the perturbation vector.
• L0: Limits the number of pixels or features that can be modified.
• Physical Constraints: For real-world attacks, includes robustness to lighting, angle, and printability (e.g., adversarial patches).
• Query Budget: In black-box settings, limits the number of API calls allowed.

Identifies the entry points where an adversary can interact with the ML system. This dimension maps the pipeline stages vulnerable to manipulation.

• Training-Time: Attacker poisons the dataset or modifies pre-trained weights before deployment. Includes backdoor injection and data poisoning.
• Inference-Time: Attacker crafts adversarial examples at prediction time. The model is frozen, but inputs are manipulated.
• Supply Chain: Compromise of third-party models, libraries, or datasets before integration.
• API Exploitation: Query-based attacks that extract model behavior or training data through the inference endpoint.

Explicitly states what the defender can and cannot do, establishing the baseline for security guarantees. This prevents evaluating defenses under unrealistic conditions.

• Data Sanitization: Can the defender filter or clean training data? If not, poisoning defenses must be robust to raw inputs.
• Model Access: Is the model publicly deployed or gated behind authentication?
• Detection Capability: Can the defender detect attacks in real-time, or must the model be inherently robust?
• Retraining Frequency: How often can the model be updated to patch vulnerabilities?
• Gradient Obfuscation: Explicitly note if gradients are hidden—this is often a brittle defense (gradient masking) that fails against black-box transfer attacks.