Model Extraction Attack

The attacker sends thousands to millions of carefully selected inputs to the victim's API and records the corresponding outputs—class probabilities, logits, or hard labels. These input-output pairs form a synthetic labeled dataset used to train a surrogate model that approximates the target's decision boundary. The attack exploits the fundamental property that model predictions encode learned knowledge, making each API response a leakage vector. Attackers often use active learning strategies to select queries near the decision boundary, maximizing information gain per query while minimizing detection risk.

For models with known architectures and differentiable activation functions, extraction can be formulated as a system of equations. Given a model with n parameters, an attacker can solve for exact weights by querying the model on n+1 carefully chosen inputs and observing the outputs. This is particularly effective against logistic regression, shallow neural networks, and support vector machines where the relationship between inputs and outputs is mathematically tractable. The attack recovers parameters to within numerical precision limits, producing a bitwise-identical copy in some cases.

Rather than recovering exact parameters, this variant aims to faithfully reproduce the target model's classification surface. The attacker queries points throughout the input space and uses the responses to train a surrogate that matches the target's predictions on any arbitrary input with high fidelity. The fidelity metric—typically agreement rate on held-out test data—measures how well the surrogate replicates the victim's behavior. High-fidelity extraction (above 95% agreement) effectively steals the model's functional value even without architectural knowledge.

A practical extraction variant where the attacker does not need the exact target architecture. Instead, they train any high-capacity model on the stolen query-response pairs, producing a knockoff that matches or even exceeds the victim's performance on the original task. This is particularly dangerous because:

• The attacker can use off-the-shelf architectures like ResNet or transformers
• The knockoff may generalize better than the victim if trained with more data augmentation
• The attack requires no insider knowledge of the target system

When APIs return full confidence vectors rather than just top-1 labels, extraction becomes dramatically more efficient. Each query leaks logit-level information about the model's internal representations, providing richer supervision for surrogate training. Attackers can reconstruct decision boundaries with orders of magnitude fewer queries compared to label-only access. This is why security best practices recommend returning only hard labels or top-k predictions and applying output perturbation to obscure precise confidence values.

Defenders can detect extraction attacks by monitoring query pattern anomalies:

• High query volume from a single source
• Systematic input space coverage rather than natural distribution sampling
• Near-boundary query clustering indicative of active learning strategies
• Sequential query dependencies where each input builds on previous responses

Detection systems use statistical hypothesis testing on query distributions and rate limiting to make large-scale extraction economically infeasible.

A privacy attack that reconstructs sensitive training data by exploiting access to a model's confidence scores or gradients. Unlike extraction, which steals the model functionality, inversion steals the underlying data. An attacker queries the model and uses the output probabilities to iteratively reconstruct a prototype of a target class, such as a recognizable face from a facial recognition system.

A privacy attack that determines whether a specific data record was part of the model's training set. Attackers exploit the tendency of models to be overconfident on memorized training examples. This is often a precursor to extraction or inversion, as confirming membership validates the surrogate model's fidelity. Defenses include differential privacy and limiting query precision.

An attack methodology executed with zero knowledge of the target model's architecture, parameters, or training data. The adversary relies solely on input-output query access. Model extraction is a specific goal within the black-box paradigm, where the attacker uses the observed predictions as a labeled dataset to train a surrogate model. Transferability of adversarial examples often enables black-box evasion.

A formal mathematical framework that provides a provable guarantee against membership inference and data reconstruction. By adding calibrated statistical noise to query outputs or during training, differential privacy ensures that the presence or absence of any single training record is statistically indistinguishable. This directly limits the fidelity of extracted surrogate models by degrading the precision of stolen decision boundaries.

A defensive technique that augments the training dataset with adversarially perturbed examples labeled with the correct class. While primarily a defense against evasion attacks, it can indirectly hinder extraction by forcing the model to learn a flatter loss landscape. A surrogate model trained on queries from a robust model often fails to capture the true decision boundary, reducing extraction fidelity.

The legitimate, intended transfer of knowledge from a large teacher model to a smaller student model by training the student on the teacher's soft output probabilities. Model extraction is functionally identical to distillation but performed by an unauthorized adversary. Defenses against extraction must carefully distinguish between legitimate distillation for edge deployment and malicious theft of intellectual property.