Model Inversion Attack

A binary classification attack where an adversary determines whether a specific record x was present in the model's training set D. By analyzing the model's prediction confidence, loss values, or gradient norms on x, the attacker exploits the model's tendency to be overconfident on memorized training examples. Shadow models—copies trained on synthetic data mimicking the target's distribution—are commonly used to train the attack classifier. This is a direct precursor to model inversion, as confirming membership validates that extracted features correspond to real individuals.

A mathematical framework providing a provable upper bound on information leakage from a computation. A randomized mechanism M satisfies (ε, δ)-differential privacy if for any two neighboring datasets D and D' differing by one record, the output distributions are nearly indistinguishable. Key mechanisms include:

• Gaussian Mechanism: Adds noise calibrated to L2 sensitivity
• Laplace Mechanism: Adds noise calibrated to L1 sensitivity
• Privacy Budget (ε): Smaller epsilon enforces stronger guarantees, directly limiting the fidelity of any model inversion reconstruction.

A preprocessing step in differentially private SGD that bounds the influence of any single training example on the model update. Each per-example gradient g_i is scaled down if its L2 norm exceeds a threshold C: g_i = g_i * min(1, C / ||g_i||_2). This limits the sensitivity of the gradient computation, which is the maximum change in the output caused by adding or removing one record. Without clipping, outlier data points would require excessive noise injection, destroying utility. Clipping is a critical defense against gradient leakage-based inversion.

A cryptographic protocol ensuring a central server learns only the sum of client model updates, never individual contributions. Using Shamir secret sharing and pairwise masking, each client i adds a random mask r_i to its update such that all masks cancel upon summation: Σ(w_i + r_i) = Σw_i. This prevents an honest-but-curious aggregator from performing gradient inversion attacks on individual updates. Often combined with Trusted Execution Environments (TEEs) for hardware-rooted integrity of the aggregation logic.

An integrity attack where an adversary injects malicious samples into the training dataset to corrupt model behavior. In the context of model inversion, backdoor poisoning is particularly dangerous: the attacker implants a trigger pattern that, when present, causes the model to confidently misclassify or reveal private features. A poisoned model may learn a direct mapping from the trigger to a specific training example's representation, effectively creating a built-in inversion mechanism. Defenses include robust aggregation and spectral anomaly detection on gradient updates.

A cryptographic primitive enabling computation directly on ciphertext. In Fully Homomorphic Encryption (FHE), an operation f on encrypted data Enc(x) yields Enc(f(x)) without ever decrypting. For federated learning, this allows the aggregation server to compute Enc(Σw_i) from individually encrypted client updates. While computationally intensive, FHE provides the strongest theoretical defense against model inversion during aggregation, as the server never accesses plaintext gradients. Partially Homomorphic Encryption (PHE) offers a practical trade-off for additive aggregation only.