Gradient Leakage

The cardinality of the input batch is the single most critical factor. Gradient leakage is trivially solved for a batch size of 1, as the gradient is a deterministic function of a single input.

• Small Batches (n ≤ 8): High-fidelity reconstruction is often achievable.
• Large Batches: The aggregated gradient represents an average, making it impossible to separate individual contributions without auxiliary information.
• Duplicate Samples: If a batch contains identical data points, their gradient signals constructively interfere, making them easier to extract.

The network topology dictates how much positional information is preserved in the gradient. Shallow networks and specific layer types leak more spatial structure.

• Fully Connected Layers: Leak heavily because the gradient w.r.t. the weight matrix is the outer product of the input and the upstream gradient.
• Convolutional Layers: Leak local patterns; gradients w.r.t. inputs can reveal edge and texture information.
• Transformer Attention: The attention map gradients can expose token-to-token relationships, enabling sequence reconstruction.
• Skip Connections: Residual links propagate input signals deeper into the network, providing stronger leakage signals in later layers.

The choice of non-linearity determines the gradient flow and the invertibility of the forward pass. Functions with flat regions destroy information.

• ReLU: Sparse gradients. Dead neurons (negative pre-activations) produce zero gradients, creating irreversible information loss for those input regions.
• Sigmoid/Tanh: Saturating regions produce vanishingly small gradients, making reconstruction numerically unstable.
• Linear/Identity: Zero information loss; the gradient is a direct linear transformation of the input.
• Leaky ReLU/GeLU: Provide non-zero gradients for negative inputs, preserving more information than standard ReLU.

The phase of training significantly impacts leakage fidelity. Gradients carry different information at initialization versus convergence.

• Early Training (High Loss): Gradients have large magnitudes and are dominated by the error signal, making input reconstruction highly accurate.
• Near Convergence (Low Loss): Gradients approach zero, and the signal-to-noise ratio collapses. Reconstruction becomes numerically infeasible.
• Gradient Clipping: Artificially bounding gradient norms destroys the precise magnitude information needed for analytical inversion.

The ratio of parameters to input pixels determines if the reconstruction problem is over-determined or under-determined.

• Low-Resolution Images (e.g., CIFAR): The number of gradient parameters vastly exceeds the input pixels, making the inversion problem highly over-determined and easy to solve.
• High-Resolution Images (e.g., HD Medical Scans): The input space is massive. Reconstruction requires solving an under-determined system, relying heavily on natural image priors.
• Discrete Text Tokens: Gradient leakage on transformer embeddings requires solving a discrete optimization problem to map continuous gradients back to a specific vocabulary token.

The optimizer choice alters the raw gradient before transmission. Stateful optimizers leak historical information.

• SGD: Transmits the pure stochastic gradient, providing the cleanest signal for an attacker.
• Adam/AdamW: Transmits a bias-corrected moving average of the gradient. The momentum terms act as a low-pass filter, smoothing out high-frequency details but leaking a temporal average of past inputs.
• Local Steps: Multiple local updates before sharing aggregate the gradient over several batches, creating a complex, time-averaged signal that is harder to invert than a single-step gradient.

A class of privacy attacks where an adversary analyzes a trained model's outputs or shared gradients to reconstruct representative samples of the private training data. Unlike gradient leakage—which exploits intermediate updates—model inversion typically targets the final deployed model. Attackers iteratively optimize a random input to maximize the model's confidence on a target class, revealing class-level prototypes of sensitive data. In healthcare, this could reconstruct a canonical patient face from a diagnostic model trained on facial phenotype datasets.

A cryptographic protocol that allows a central server to compute the sum of model updates from multiple clients while ensuring the server cannot inspect any individual client's contribution in plaintext. This directly mitigates gradient leakage by preventing the server from accessing raw gradients. The protocol uses secret sharing where each client masks its update with pairwise random seeds that cancel out during summation. Only the aggregate is revealed, making single-client reconstruction attacks infeasible.

A mathematical framework that injects calibrated statistical noise into model updates to provide a provable guarantee that any single individual's record cannot be inferred. Applied to federated learning, DP bounds the information leaked through gradients by clipping update norms and adding Gaussian noise. The privacy budget epsilon (ε) quantifies the guarantee—lower values mean stronger privacy but reduced model utility. DP-SGD is the standard algorithm for training with differential privacy guarantees.

An adversarial attack where a malicious participant injects corrupted or mislabeled data into the federated training process to degrade model performance or install a backdoor trigger. While gradient leakage is a confidentiality threat, data poisoning is an integrity threat. Attackers can craft gradients that appear benign but cause targeted misclassification. Defenses include robust aggregation rules like Krum or trimmed mean, which filter out outlier updates before averaging.

A cryptographic scheme enabling computation directly on encrypted data, producing an encrypted result that matches the output of operations on plaintext. In federated learning, HE allows the server to aggregate encrypted model updates without ever seeing the raw gradients. This eliminates the gradient leakage attack surface entirely. The trade-off is significant computational overhead—fully homomorphic encryption can increase computation by orders of magnitude, making it practical primarily for cross-silo settings with powerful servers.

A hardware-enforced isolated area within a main processor that guarantees confidentiality and integrity of code and data loaded inside it. TEEs like Intel SGX or AMD SEV can protect gradient computation from the host operating system and cloud provider. In federated learning, TEEs provide a hardware root of trust for secure aggregation, ensuring gradients are processed in an encrypted memory enclave invisible to the server's own software stack. This defends against gradient leakage without the overhead of cryptographic approaches.