Vertical FL with Differential Privacy

This mechanism adds calibrated noise directly to the gradients during the vertical backpropagation phase. The noise is sampled from a distribution like the Gaussian or Laplace mechanism, scaled by the gradient's sensitivity (the maximum possible change from any single data point) and a chosen privacy budget (ε, δ).

• Key Property: Provides privacy for the model parameters during training, preventing reconstruction of raw features from the shared updates.
• Challenge: Requires careful sensitivity analysis for the specific split neural network architecture and loss function to avoid excessive noise that cripples model utility.

Privacy noise is applied to the intermediate outputs (activations) at the cut layer before they are sent from a feature owner to the label owner. This protects the raw input features since the intermediate values are a deterministic function of them.

• Implementation: The feature owner computes the forward pass locally, then adds noise to the resulting vector before transmission.
• Advantage: Can be more efficient than gradient perturbation in some architectures, as noise is added once per forward pass rather than during each parameter update.
• Consideration: The sensitivity of the intermediate outputs must be bounded, often requiring techniques like gradient clipping on the upstream network portion.

Applied during the entity alignment phase using protocols like Private Set Intersection (PSI). Each party perturbs their local identifiers (e.g., user IDs) with LDP mechanisms before engaging in the alignment protocol.

• Function: Protects the membership of an entity in a party's dataset. An adversary cannot confidently determine if a specific user's data was used for alignment.
• Mechanism: Often uses randomized response techniques, where an identifier is reported truthfully with probability p and replaced with a random value otherwise.
• Benefit: Provides a privacy guarantee at the very first step of VFL, before any model training begins.

A complete VFL-DP training run involves multiple privacy mechanisms applied over many rounds. Composition theorems (sequential, parallel, advanced) are used to track the total privacy loss.

• Core Concept: The overall privacy guarantee (ε_total, δ_total) accumulates across training iterations and across different perturbed values (gradients, outputs).
• Engineering Task: A privacy budget is allocated per round and per party. Common strategies include:
  • Fixed Budget: Evenly split the total budget across all training rounds.
  • Adaptive Budget: Use more budget in early rounds for faster convergence, less in later rounds for fine-tuning.
• Tool: The Moments Accountant or GDP (Gaussian Differential Privacy) composition provides tighter bounds than basic sequential composition.

Combines cryptographic secure aggregation with differential privacy. Feature owners first add DP noise to their local updates (gradients/intermediate outputs). These noisy updates are then encrypted and aggregated using a protocol like SecAgg so the server only sees the summed, already-noisy result.

• Privacy Amplification: Secure aggregation can provide privacy amplification by sampling. If the secure aggregation protocol randomly selects a subset of clients, the effective privacy guarantee can be stronger than the local DP guarantee of each client.
• Defense-in-Depth: This creates a dual-layer protection: DP prevents leakage from the aggregated result, while cryptography prevents the server from inspecting any individual client's noisy update.

The core engineering challenge in VFL-DP is balancing the privacy-utility trade-off. Key hyperparameters directly control this balance:

• Noise Scale (σ): The standard deviation of the Gaussian noise. Larger σ increases privacy (lower ε) but reduces model accuracy.
• Clipping Norm (C): Bounds the L2 norm of gradients or intermediate outputs. This limits sensitivity, allowing for less noise to achieve the same DP guarantee. Incorrect clipping can destabilize training.
• Sampling Rate (q): The fraction of data used per training round. Lower rates often improve the privacy guarantee due to privacy amplification.
• Optimizer Choice: Adaptive optimizers like Adam can sometimes be more robust to the noise introduced by DP than plain SGD.

Media companies, retailers, and ad platforms possess different user attributes (content engagement, purchase history, demographic segments). VFL with DP allows for the training of a unified recommendation engine that leverages all these signals without building a central data lake, avoiding privacy violations and antitrust concerns.

• Workflow: A streaming service (label owner with watch/click data) collaborates with a retail partner (feature owner with purchase data) to predict user preferences.
• DP Application: The retailer's local model updates are clipped to a fixed norm (L2 sensitivity bounding) and perturbed with noise before secure aggregation. This prevents the streaming service from inferring exact purchase items.
• Advantage: Achieves high personalization and ad relevance while providing users with a provable privacy guarantee, enhancing brand trust and regulatory standing.

Different municipal departments and utility companies manage separate sensor networks (traffic cameras, air quality monitors, smart meter data) pertaining to the same geographic zones and time periods. VFL with DP enables joint analytics for urban planning and anomaly detection without merging sensitive infrastructure data.

• Use Case: Predicting traffic congestion by combining features from the transportation department's camera feeds (vehicle count) and the environmental agency's weather sensors (precipitation, visibility).
• DP Implementation: Each department's local model acts as a feature extractor. The embeddings it sends to the central traffic model are noised using the Gaussian mechanism, ensuring individual sensor readings cannot be reverse-engineered.
• Value: Optimizes public resource allocation and improves predictive maintenance schedules while protecting operational security details of critical infrastructure.

Multiple entities in a supply chain (raw material supplier, manufacturer, logistics provider) hold different features about the same products or components (material specs, production line telemetry, shipping conditions). VFL with DP facilitates predictive models for quality control, demand forecasting, and logistics without exposing proprietary operational data.

• Scenario: Predicting component failure by combining the supplier's material composition data, the factory's assembly line vibration sensors, and the shipper's temperature logs.
• Privacy Technique: A differentially private stochastic gradient descent (DP-SGD) variant is used during the vertical backpropagation phase. Gradients from each party are clipped and noised, providing immunity to data reconstruction attacks by other consortium members.
• Impact: Reduces downtime, optimizes inventory, and improves product reliability through collaborative intelligence, while fiercely protecting each company's trade secrets and competitive advantage.

Different government agencies (tax, social services, law enforcement) maintain separate databases on citizens and businesses for specific statutory purposes. VFL with DP allows for limited, privacy-guaranteed collaboration to identify systemic risks, fraud, or policy impacts without creating unconstitutional mass surveillance databases.

• Application: Detecting complex public benefits fraud by aligning entities across tax records (income features) and social services databases (benefit claim labels).
• DP Guarantee: A formal privacy budget (ε, δ) is established and tracked per analysis. Noise addition ensures the output of the federated model does not significantly change if any single individual's record is added or removed from any agency's database.
• Significance: Enhances governmental efficiency and effectiveness in service delivery while embedding algorithmic transparency and privacy-by-design into inter-agency cooperation, as mandated by modern digital governance frameworks.

Differential Privacy (DP) is a rigorous mathematical framework that quantifies and bounds the privacy risk to any individual record from participating in a computation. It provides a formal guarantee that the presence or absence of any single data point in the training dataset has a negligible impact on the algorithm's output.

• Core Mechanism: Achieved by injecting carefully calibrated random noise (e.g., from a Laplace or Gaussian distribution) into the computation.
• Privacy Budget (ε, δ): The parameters ε (epsilon) and δ (delta) define the strength of the guarantee. A smaller ε offers stronger privacy but typically reduces model utility.
• Application in FL: In federated learning, DP is commonly applied to client updates (gradients) before they are sent to the server for aggregation, ensuring the server cannot infer specifics about any client's local dataset.

Secure Multi-Party Computation (MPC) is a subfield of cryptography that enables a group of distrusting parties to jointly compute a function over their private inputs while revealing nothing but the final output. In VFL, MPC protocols are used to perform operations like gradient calculation and aggregation without exposing raw features or intermediate values.

• Contrast with DP: While DP provides statistical privacy guarantees against an untrusted aggregator, MPC provides cryptographic security guarantees against colluding parties, assuming they follow the protocol.
• Common Techniques: Uses secret sharing or garbled circuits to distribute computation. For example, a feature owner can secret-share its intermediate output, allowing the label owner to compute gradients without learning the actual value.
• Hybrid Approach: VFL systems often combine MPC (for secure computation) with DP (for an additional layer of privacy against potential protocol failures or future attacks).

Homomorphic Encryption (HE) is a form of encryption that allows specific types of computations to be performed directly on ciphertext, generating an encrypted result that, when decrypted, matches the result of the operations performed on the plaintext. In VFL, HE enables the label owner to perform computations on encrypted intermediate outputs from feature owners.

• Workflow: A feature owner encrypts its intermediate output using the label owner's public key and sends the ciphertext. The label owner performs model computations (e.g., applying its part of the neural network) on the ciphertext, returning an encrypted gradient. The feature owner can then decrypt this result with its private key.

• Key Challenge: Computational Overhead. HE operations are orders of magnitude slower than plaintext computations, making it a major contributor to vertical computation overhead.

• Partial vs. Full HE: Many VFL implementations use partial homomorphic encryption (e.g., Paillier), which supports only addition or multiplication, as it is more efficient than fully homomorphic schemes.

Private Set Intersection (PSI) is a foundational cryptographic protocol for VFL that allows two or more parties to discover the common identifiers (e.g., user IDs) in their datasets without revealing any identifiers that are not in the intersection. This is the critical first step of entity alignment.

• Purpose: Enables parties to align their vertically partitioned data on the same set of entities (rows) before training begins, without leaking which entities they hold.
• Example: A bank (with credit features) and an e-commerce company (with purchase history) can use PSI to find their shared customers without the bank learning the e-commerce company's full customer list, and vice versa.
• Protocol Types: Modern PSI protocols, such as those based on oblivious transfer or Bloom filters, are highly efficient, capable of computing the intersection of millions of records in seconds.

A Split Neural Network is the standard model architecture for Vertical Federated Learning, where a single neural network is partitioned into multiple segments hosted by different parties. The division point is called the cut layer.

• Architecture:
  • Feature-Owner Segment: The initial layers of the network, which process the local raw features. The output of these layers is the intermediate output or embedding.
  • Label-Owner Segment: The remaining layers of the network, which take the intermediate outputs from all feature owners, combine them, and produce the final prediction or loss.
• Training Flow:
  1. Vertical Forward Pass: Feature owners compute to the cut layer and send intermediate outputs to the label owner.
  2. Loss & Gradient Calculation: The label owner completes the forward pass, computes the loss, and calculates gradients for its segment.
  3. Vertical Backward Pass: Gradients are propagated back to the cut layer and sent to each feature owner, which then updates its local segment.

Privacy-Preserving Aggregation encompasses the techniques used to combine model updates from multiple parties in FL without revealing any individual contribution. In VFL, this often applies to aggregating intermediate outputs or gradients from multiple feature owners.

• Secure Aggregation: A specific cryptographic protocol (often using MPC or masking techniques) that allows a server to compute the sum of client updates without being able to inspect any single client's update. This protects against a curious aggregator.
• Integration with DP: Differential privacy is frequently applied in conjunction with secure aggregation. Noise can be added locally by each client (Local Differential Privacy) or by the server after secure aggregation (Central Differential Privacy).
• Vertical Context: In VFL, aggregation might involve securely summing gradients from feature owners for the shared cut layer parameters or combining multiple encrypted intermediate outputs before further processing by the label owner.