Vertical Gradient Computation

The model is physically divided into segments, or sub-models, owned by different parties. A feature owner holds the bottom layers that process its local features. The label owner typically holds the top layers, culminating in the output and loss calculation. The cut layer is the designated boundary where intermediate outputs are exchanged. This architectural split is the foundation that enables distributed forward and backward passes without sharing raw data.

Each feature owner performs a local forward propagation on its segment of the model using its private features. The output of this computation, known as the intermediate output or embeddings, is produced at the cut layer. This intermediate output is then securely transmitted (often in encrypted form) to the label owner. The label owner receives these embeddings from all feature owners, concatenates them, and completes the forward pass through its top model segment to compute the final prediction and loss.

This is the core of vertical gradient computation. The label owner calculates the initial gradients from the loss function. These gradients are then backpropagated through its top model segment. At the cut layer, the label owner computes the gradients with respect to the received intermediate outputs. These gradients are sent back to the corresponding feature owners. Each feature owner then uses these received gradients to perform local backpropagation through its own model segment, calculating the gradients for its private model parameters and features.

The exchange of intermediate outputs and gradients constitutes the primary communication overhead and privacy risk. To prevent data reconstruction attacks, these values must be protected. Common techniques include:

• Homomorphic Encryption (HE): Allows computations (like the label owner's forward pass) to be performed directly on encrypted intermediate outputs.
• Secure Multi-Party Computation (MPC): Uses cryptographic protocols to jointly compute gradients without revealing private inputs.
• Differential Privacy (DP): Adds carefully calibrated noise to the communicated tensors to provide a mathematical privacy guarantee.

After completing the distributed backward pass, each party holds the gradients for its own segment of the model. Each party then performs a local optimization step (e.g., Stochastic Gradient Descent) to update its private model parameters. Critically, the raw model parameters themselves are never shared or aggregated across parties, unlike in Horizontal Federated Learning. The global model is defined implicitly by the collective state of all the local model segments, which are updated in a coordinated fashion through the gradient flow.

Vertical gradient computation is governed by a strict training protocol that sequences all steps. A central coordinator (often the label owner) orchestrates the rounds:

1. Initiation: Broadcasts the start of a new training round.
2. Forward Coordination: Collects encrypted intermediate outputs from feature owners.
3. Gradient Distribution: Computes and sends back the relevant gradients.
4. Synchronization: Ensures all parties update their models in lockstep. Frameworks like FATE and PyVertical implement these protocols, handling communication, encryption, and fault tolerance.

A split neural network is the model architecture used in VFL where the network is divided into segments, each residing on a different party. The feature owners hold the initial layers that process their local features, while the label owner typically holds the later layers leading to the final output. The division point is defined by the cut layer.

These are the two phases of distributed gradient computation in VFL.

• Vertical Forward Propagation: Each feature owner computes the forward pass on its local model segment and sends the encrypted intermediate output to the next party (e.g., the label owner) to complete the forward pass and calculate the loss.
• Vertical Backpropagation: The label owner computes the initial gradients and passes them back through the network. Each party receives gradients for its segment, computes local parameter updates, and may send gradients back to previous parties, all without exposing raw data.

Secure aggregation is a cryptographic protocol for combining model updates from multiple parties without revealing any individual contribution. In VFL, this is often applied to gradients or intermediate outputs. Common techniques include:

• Masking with Secret Sharing: Clients add a random mask to their update; masks cancel out when aggregated by the server.
• Homomorphic Encryption: Allows the server to perform arithmetic on encrypted updates. This prevents a curious aggregator from inferring sensitive information from a single client's update.

Homomorphic Encryption is a form of encryption that allows computations to be performed directly on ciphertext. In VFL, HE enables a party (often the label owner) to perform calculations on encrypted intermediate results received from feature owners. For example, the label owner can compute the loss and start backpropagation on encrypted data, preserving the confidentiality of the feature owner's intermediate outputs throughout the gradient computation process.

Secure Multi-Party Computation is a cryptographic suite that enables multiple parties to jointly compute a function over their private inputs without revealing those inputs to each other. In Vertical MPC, this framework is used to implement the entire vertical gradient computation protocol. Parties collaboratively calculate forward passes, loss, and gradients using secret-shared values, providing strong privacy guarantees often at the cost of higher communication and computation overhead compared to simpler masking techniques.

The vertical training protocol is the complete, step-by-step specification governing a single training round in a VFL system. It defines:

• The sequence of communication between feature and label owners.
• The exact format of exchanged messages (e.g., encrypted embeddings, gradients).
• The cryptographic operations required at each step.
• The aggregation and update rules. Protocols like SecureBoost for trees or those for split neural networks ensure that the vertical gradient computation is executed correctly, efficiently, and privately.