Model Extraction

The adversary sends carefully selected inputs to the victim's prediction API and records the returned outputs—class labels, confidence scores, or logits. These input-output pairs form a labeled dataset that trains a substitute model to mimic the original.

• Equation extraction: Recovers the mathematical function f(x) ≈ y
• High-fidelity cloning: Achieves 90%+ agreement with the target model
• No internal access required: Operates entirely through the public API boundary

The attacker exploits the fundamental property that every API response leaks information about the model's internal decision surface.

Sophisticated extraction attacks use uncertainty sampling to maximize information gain per query. Rather than random inputs, the attacker selects points near the decision boundary where the model is least confident.

• Margin sampling: Queries inputs where top two class probabilities are closest
• Jacobian-based heuristics: Targets regions of high gradient sensitivity
• Adaptive synthesis: Generates synthetic queries that progressively refine the replica

These strategies dramatically reduce the query budget needed for successful extraction, often requiring orders of magnitude fewer queries than passive random sampling.

APIs that return full confidence vectors rather than only hard labels are significantly more vulnerable. Each confidence score reveals the model's internal certainty distribution across all classes.

• Rich gradient signal: Confidence scores expose the shape of the decision surface
• Label-only hardening: Returning only the top-1 class substantially increases extraction difficulty
• Logit access: Raw logits provide the strongest signal for replica training

The granularity of API output directly determines extraction vulnerability. Defenders should minimize information leakage by returning only essential predictions.

A successfully extracted model enables competitors to replicate years of research investment without the associated costs of data collection, architecture design, and hyperparameter tuning.

• Model commoditization: Proprietary models become replicable commodities
• Training data inference: Extracted models can leak information about private training data
• Bypass licensing: Circumvents per-query pricing and usage restrictions
• Adversarial reconnaissance: Stolen replicas enable white-box attack development against the original

For ML-as-a-Service providers, extraction represents both IP theft and a direct revenue threat.

Defending against extraction requires balancing model utility against information leakage. Key strategies include:

• Rate limiting: Caps on queries per user and per time window
• Query monitoring: Detecting anomalous query patterns indicative of extraction
• Output perturbation: Adding calibrated noise to confidence scores
• Differential privacy: Formal guarantees limiting information per query
• Watermarking: Embedding verifiable ownership markers in model outputs

No single defense is sufficient. Effective protection requires layered controls across the API boundary, monitoring infrastructure, and model architecture.

Beyond functionality cloning, attackers can infer architectural details through side-channel analysis of query timing, memory patterns, and output structure.

• Architecture fingerprinting: Output dimensionality reveals layer widths
• Timing side-channels: Inference latency exposes model depth and complexity
• Transfer attacks: Adversarial examples crafted on the replica transfer to the original

The extracted replica serves as a surrogate for developing further attacks, including adversarial examples and membership inference, against the original production model.

A maliciously perturbed input designed to cause a machine learning model to make a mistake while appearing unmodified to human observers. These inputs exploit the model's decision boundaries by adding imperceptible noise.

• Key Mechanism: Adds small perturbations in the direction of the loss gradient
• Real-World Impact: Can cause autonomous vehicles to misclassify stop signs as speed limit signs
• Relationship to Extraction: Often used as a precursor to probe decision boundaries before cloning

An attack that corrupts the training dataset by injecting malicious samples, causing the model to learn a backdoor or degrade its overall performance. Unlike extraction attacks that steal the model, poisoning compromises the model at its foundation.

• Attack Vector: Compromised data sources or insider threats
• Defense Strategy: Data provenance tracking and anomaly detection on training pipelines
• Critical Distinction: While extraction steals IP, poisoning destroys model integrity from within

A privacy attack that reconstructs sensitive features or training data samples from a model's parameters or outputs. This attack exploits the model's memorization of training data rather than stealing the model architecture itself.

• Risk Surface: Models trained on facial recognition or medical data
• Mitigation: Differential Privacy limits individual training point influence
• Synergy with Extraction: Extracted substitute models often exhibit higher inversion vulnerability

An attack that determines whether a specific data record was used to train a target machine learning model. This represents a severe privacy violation, particularly for models trained on sensitive personal or proprietary data.

• Technique: Analyzes confidence scores and loss values for target samples
• Defense: Differential Privacy Defense provides formal guarantees against inference
• Extraction Connection: Extracted models can amplify membership leakage from the original model

An adversarial attack that relies solely on querying a model's outputs without any access to its internal parameters or architecture. This is the primary attack model for extraction, as most commercial APIs expose only input-output pairs.

• Capability: Attacker can only observe predictions and confidence scores
• Extraction Method: Uses query-response pairs to train a substitute model
• Real-World Example: Extracting a cloud-hosted NLP model via its public prediction API