Model Extraction Attack

The adversary sends a stream of carefully selected inputs to the victim model and records the corresponding outputs. By training a substitute model on these input-output pairs, the attacker approximates the decision boundary of the target. In healthcare federated learning, an attacker could reconstruct a proprietary diagnostic classifier by querying it with synthetic patient data and learning from the returned predictions. Equation-based extraction targets models that expose confidence scores, while label-only extraction works even when only hard class labels are returned, requiring more sophisticated boundary-tracing techniques.

For models with known architectures and differentiable activation functions, an attacker can formulate the extraction as a system of equations. Each query provides constraints on the model's internal weights and biases. By solving these equations analytically or through optimization, the adversary recovers the exact numerical parameters. This is particularly dangerous for logistic regression and shallow neural networks where the number of unknown parameters is small relative to the query budget. In federated medical imaging, a stolen radiology model's weights could reveal diagnostic heuristics developed over years of proprietary research.

Models that return full probability vectors rather than just class labels leak significantly more information per query. The confidence scores reveal the precise distance to the decision boundary, enabling gradient-free optimization attacks. An attacker can use hill-climbing on the confidence landscape to efficiently map the model's internal geometry. Mitigation strategies include confidence thresholding, where only top-k probabilities are returned, and temperature scaling to flatten the output distribution. In clinical decision support systems, raw confidence scores could expose the relative importance the model assigns to specific biomarkers.

Sophisticated extraction attacks employ active learning to minimize the number of queries needed. Rather than random sampling, the attacker selects queries that maximize information gain about the decision boundary. Techniques include:

• Uncertainty sampling: querying points where the substitute model is least confident
• Query synthesis: generating adversarial examples that lie near the estimated boundary
• Adaptive sampling: iteratively refining the substitute model and selecting queries that resolve remaining ambiguities These strategies make extraction feasible even under strict rate limiting, as each query is maximally informative.

Defenses against model extraction operate on multiple layers:

• Query rate limiting and per-user query budgets restrict the adversary's information channel
• Differential privacy during inference adds calibrated noise to outputs, bounding information leakage per query with a provable epsilon guarantee
• Out-of-distribution detection identifies anomalous query patterns that deviate from legitimate clinical data distributions
• Watermarking embeds verifiable ownership signatures that persist even in extracted copies
• Round-trip detection monitors for queries that appear to be probing decision boundaries systematically rather than representing genuine patient cases

In federated learning, the attack surface expands beyond the central model to include intermediate gradient updates and local client models. An adversary posing as a legitimate hospital node can:

• Extract the global model by participating in enough training rounds and observing aggregated updates
• Exploit gradient leakage to reconstruct not just the model but also private training data
• Use model inversion on the extracted model to recover sensitive patient features
• Deploy Sybil identities to amplify extraction bandwidth across multiple fake client nodes The distributed trust model of federated learning makes robust identity verification and contribution auditing essential.

An adversarial technique to determine whether a specific data record was part of a model's training dataset. This creates a direct privacy violation, especially in healthcare contexts.

• Exploits differences in model confidence on seen vs. unseen data
• Can reveal if a patient's record was used in a clinical trial model
• Often used as a precursor to more sophisticated extraction attacks

A training-time attack that corrupts model integrity by injecting malicious samples into the training data. In federated settings, a compromised client can poison the global model through its local updates.

• Availability poisoning degrades overall model accuracy
• Targeted poisoning causes misclassification on specific inputs
• Federated learning's distributed nature increases the attack surface

A model's quantified resilience against intentionally crafted inputs designed to cause misclassification. Robustness is measured by the minimum perturbation required to change a prediction.

• Evasion attacks add imperceptible noise to inputs at inference time
• Certified robustness provides mathematical guarantees against perturbation
• Extraction attacks often use adversarial queries to probe decision boundaries

An attack that reconstructs private local training data from shared model gradients during collaborative learning. This is the federated learning analog of model inversion.

• Reconstructs pixel-level images from gradient updates
• Exploits the fact that gradients encode information about training samples
• Secure aggregation and differential privacy are primary defenses

A technique for embedding a unique, verifiable identifier into a model to assert intellectual property ownership. Watermarking is the defensive counterpart to model extraction.

• Embeds trigger sets that produce specific outputs only for the owner
• Survives fine-tuning and model compression attempts
• Provides legal evidence of theft when extraction is detected