Cosine Similarity Loss

Unlike Euclidean distance, which penalizes differences in vector magnitude, cosine similarity loss focuses exclusively on the angle between two vectors. It is computed as 1 - cos(θ), where cos(θ) is the dot product of the normalized vectors. In gradient leakage, this allows the attack to prioritize matching the directional pattern of the real gradient, ignoring discrepancies in scale that may arise from batch size or learning rate differences.

A defining mathematical property of cosine similarity loss is its insensitivity to vector magnitude. If a synthetic gradient is a scalar multiple of the real gradient, the cosine loss is zero. This is critical in federated learning attacks because the adversary often cannot know the exact scaling factor applied by the server. By optimizing for angular alignment, the attack reconstructs the structural information of the private data without needing to guess the gradient norm.

In gradient inversion attacks like Deep Leakage from Gradients (DLG), replacing Euclidean loss with cosine similarity loss often results in significantly faster convergence. This is because the loss landscape for angular optimization provides a stronger supervisory signal early in the attack. The dummy inputs rapidly rotate to match the orientation of the true gradient, leading to high-fidelity reconstructions in fewer iterations compared to magnitude-sensitive losses.

Cosine similarity loss inherently normalizes the gradient vectors to unit length before comparison. This normalization acts as a form of implicit gradient clipping, preventing exploding gradients during the attack optimization. It stabilizes the iterative reconstruction process, particularly when the real gradient contains layers with vastly different dynamic ranges (e.g., weights vs. biases), ensuring no single layer dominates the loss calculation.

While Euclidean (L2) loss minimizes the squared difference between all corresponding elements, it is highly sensitive to magnitude. In gradient inversion, this can cause the attack to waste iterations matching the scale of large-magnitude layers while ignoring the directional structure of smaller gradients. Cosine similarity loss ignores magnitude to focus purely on the orientation, which encodes the semantic content of the input data. A hybrid approach, combining both losses, is sometimes used to balance scale and direction.

Cosine similarity loss is a core objective function in modern gradient matching attacks. The attack framework iteratively updates dummy inputs and labels to minimize the cosine distance between the produced dummy gradient and the observed real gradient. This technique is particularly effective against federated learning and split learning architectures, where shared gradients or intermediate activations serve as the leakage vector for reconstructing private training samples.

A class of privacy attacks that recover sensitive input data by inverting the mathematical relationship between a model's parameters and its loss gradient. Unlike model inversion attacks that target the final trained model, gradient inversion exploits intermediate gradient updates shared during distributed training. Variants include recursive reconstruction across multiple time steps and analytical recovery for fully connected layers.

A defense technique that selectively transmits only a subset of gradient elements during distributed training to reduce the information available for leakage attacks. By communicating only the top-k gradient elements by magnitude, pruning creates an incomplete signal that significantly degrades reconstruction quality while maintaining model convergence. Often combined with gradient quantization for additional protection.

A defense mechanism that bounds the L2 norm of individual gradients before sharing, limiting the signal-to-noise ratio available to an adversary. By capping gradient magnitudes to a threshold C, clipping prevents outlier gradients from encoding excessive information about their source data. This technique serves dual purpose: it stabilizes training and forms the foundation of DP-SGD's privacy analysis.