Model Extraction Attack

The attacker's goal is to create a substitute model that approximates the victim's decision boundary with high fidelity. By sending carefully crafted inputs and collecting the corresponding outputs (labels or confidence scores), the adversary builds a labeled dataset. This dataset trains a surrogate model—often of similar architecture—that replicates the original's functionality without accessing its weights, architecture details, or training data. The attack exploits the model's prediction API as an oracle.

For certain model families, extraction can be mathematically exact. Attackers exploit the fact that models like logistic regression, SVMs, or shallow neural networks with known activation functions have parameters solvable via linear equations. By querying the model with a number of inputs equal to the parameter count plus one, an attacker can set up a system of equations and solve directly for the exact weights and biases. This renders the model's intellectual property completely transparent.

Models that return full confidence vectors rather than just class labels leak substantially more information per query. The probability distribution over classes reveals the model's internal certainty and the relative distances between decision boundaries. Attackers use these scores to compute gradient approximations via zero-order optimization methods, enabling them to reconstruct the loss landscape and train surrogates that match the victim's behavior with far fewer queries than label-only access would require.

Organizations protect against extraction through:

• Rate limiting: Capping queries per API key or IP address
• Prediction truncation: Returning only top-k classes instead of full probability vectors
• Differential privacy: Injecting calibrated noise into outputs to bound information leakage
• Rounding confidence scores: Reducing precision to degrade gradient estimation
• Query anomaly detection: Flagging systematic, boundary-probing query patterns
• Watermarking: Embedding verifiable ownership signals into model responses

Successful extraction attacks enable competitors to clone proprietary models without the research and compute investment. This undermines the victim's competitive moat, enables adversarial attacks on the cloned model (since white-box access is now available), and facilitates membership inference against the original training data. For models trained on proprietary datasets, extraction effectively steals both the model architecture and the distilled knowledge of the private data.

A privacy attack where an adversary reconstructs sensitive training data features by analyzing a model's outputs and confidence scores. Unlike extraction attacks that steal the model, inversion attacks steal the underlying data. Attackers iteratively query the target model and use gradient descent to generate synthetic inputs that maximize class confidence, revealing private attributes like faces or medical records.

• Exploits overconfident softmax outputs
• Mitigated by differential privacy and limiting prediction detail
• Closely related to membership inference attacks

A carefully perturbed input designed to cause a machine learning model to make a high-confidence misclassification while appearing unchanged to human observers. These inputs exploit the model's decision boundaries and are distinct from extraction attacks, though both probe model internals. Common in evasion attacks against computer vision systems.

• Generated via FGSM, PGD, or C&W attacks
• Transferable between surrogate and target models
• Defenses include adversarial training and input sanitization

A technique for embedding a verifiable, persistent identifier into a machine learning model to prove ownership and detect unauthorized extraction or distribution. Watermarks are implanted during training by introducing specific trigger patterns that produce unique, predictable outputs. When a stolen model is queried with the trigger, the watermark confirms IP theft.

• White-box watermarks embed patterns in weights
• Black-box watermarks use backdoor-style trigger sets
• Must survive fine-tuning, pruning, and distillation

A mathematical framework that provides provable privacy guarantees by injecting calibrated noise into model training or query responses. It bounds the influence of any single training example, making it statistically impossible to determine if a specific record was used. This directly frustrates both extraction and membership inference attacks by limiting the information leaked per query.

• Governed by the privacy budget parameter epsilon (ε)
• Lower epsilon = stronger privacy, reduced utility
• Implemented via DP-SGD during training or noisy query responses

A legitimate model compression technique where a smaller student model is trained to replicate the behavior of a larger teacher model using soft labels. This process is architecturally identical to model extraction but performed with authorized access. The distinction between distillation and theft lies entirely in authorization and intent.

• Uses soft targets (probability distributions) instead of hard labels
• Transfers generalization ability, not just predictions
• Defensive distillation can also harden models against adversarial examples

A frontline defense against extraction attacks that restricts the number, frequency, or diversity of API queries a single user can submit. By capping the information flow, defenders prevent adversaries from collecting the large volumes of input-output pairs needed to train a high-fidelity surrogate model.

• Enforced via token buckets, sliding windows, or adaptive throttling
• Can be combined with query auditing to detect systematic probing
• Trade-off: overly aggressive limits degrade legitimate user experience