Deep Leakage from Gradients (DLG)

The attack initializes a dummy input (random noise) and a dummy label, then computes their gradients through the same model. The loss function minimizes the L2 distance or cosine similarity between the dummy gradients and the actual shared gradients. Through iterative optimization (typically L-BFGS or Adam), the dummy input converges to the original private training data.

• Loss function: argmin ||∇θL(dummy) - ∇θL(real)||²
• Convergence: Often within 100-500 iterations
• Fidelity: Recovers pixel-accurate images and token-level text

DLG exploits a critical vulnerability: gradients of the classification layer directly encode the ground-truth label. For cross-entropy loss, the gradient with respect to the correct class logit is always negative, while all other class gradients are positive. This allows the attacker to extract labels without any additional computation before reconstructing inputs.

• Sign analysis: Correct class gradient < 0, incorrect classes > 0
• One-hot extraction: Label is trivially recovered from the sign pattern
• Impact: Eliminates the need for brute-force label guessing

While the original DLG paper focused on single-image reconstruction, the attack extends to small batches. The optimization objective sums the gradient matching loss across all batch elements. However, reconstruction quality degrades as batch size increases due to gradient averaging—the shared gradient becomes a mixture that obscures individual contributions.

• Effective range: Batch sizes 1-8 typically recoverable
• Limitation: Larger batches require auxiliary techniques like gradient separation
• Defense implication: Large batch sizes provide natural partial protection

DLG reconstruction quality is highly sensitive to the choice of optimizer and initialization strategy. The original work uses L-BFGS for its strong convergence properties on smooth objectives. Modern variants employ Adam with carefully tuned learning rates. Poor initialization can trap the optimization in local minima, producing blurry or incorrect reconstructions.

• Preferred optimizer: L-BFGS for deterministic convergence
• Initialization: Gaussian noise or constant values
• Restart strategy: Multiple random restarts improve robustness against local minima

To improve reconstruction fidelity, DLG incorporates a Total Variation (TV) regularization term in the loss function. TV penalizes high-frequency noise by minimizing the sum of absolute differences between neighboring pixels, acting as a natural image prior that produces smoother, more realistic outputs. This regularization is critical for recovering fine textures.

• TV loss: Σ|pixel_i - pixel_i+1| across spatial dimensions
• Effect: Suppresses checkerboard artifacts and noise
• Trade-off: Excessive TV weight oversmooths and loses detail

DLG does not require knowledge of the exact model architecture used during training. The attack demonstrates cross-architecture generalization—gradients from one architecture can be used to reconstruct inputs even when the attacker assumes a different architecture. This makes the attack practical in black-box federated learning scenarios where the server architecture may not be fully disclosed.

• Transferable across: CNNs, MLPs, and transformer variants
• Requirement: Attacker needs only the gradient tensor shape
• Robustness: Works even with architectural mismatches

A lightweight defense that reduces the information content of shared gradients before transmission. Techniques include:

• Gradient sparsification: transmitting only the top-k largest gradient values
• Quantization: reducing gradient precision to 8-bit or 4-bit integers
• Randomized pruning: dropping gradient elements with probability p

These methods degrade DLG reconstruction fidelity but do not provide formal privacy guarantees.

A defense that exploits the image prior used by DLG. By applying a Total Variation (TV) loss penalty during the inversion attack, the adversary's optimization fails to reconstruct fine textures.

• TV loss penalizes high-frequency noise and promotes piecewise smoothness
• Acts as a natural regularizer that protects against gradient leakage
• Can be combined with other defenses like gradient perturbation
• Does not impact model training accuracy