Intermediate Output

An intermediate output is the result of the forward pass computation up to the cut layer in a split neural network. It is generated by a feature owner using their local features and must be transmitted to the next party (typically the label owner) to continue the forward propagation. This mechanism allows the complete model to be computed without any party exposing their raw input data.

• Primary Role: Acts as the privacy-preserving substitute for raw feature data during collaborative computation.
• Analogy: Similar to a partially solved equation where one contributor passes their result to another to complete the calculation, without sharing the original numbers.

The intermediate output is designed to contain minimal information about the raw input data, forming the first line of defense in privacy-preserving machine learning. However, it is not inherently secure; its privacy characteristics depend on the model architecture and cryptographic safeguards.

• Information Leakage Risk: A deep or wide cut layer can allow for model inversion attacks or membership inference attacks, where an adversary attempts to reconstruct training samples.
• Mitigation Strategies: Techniques like homomorphic encryption (computing on encrypted outputs), secure multi-party computation (MPC), or adding differential privacy noise are used to protect the intermediate outputs during transmission and computation.

The properties of the intermediate output directly dictate the vertical training protocol and overall system design. Its size and computation location are key architectural decisions.

• Cut Layer Selection: Choosing where to split the model is a trade-off. An early cut (shallow layer) produces a small, low-dimensional output, reducing communication overhead but potentially harming model accuracy. A late cut (deep layer) may improve accuracy but increases communication cost and privacy risk.
• Computation/Communication Trade-off: The volume of data in the intermediate output is the primary driver of vertical communication overhead. Optimizing this is crucial for practical deployment.

During vertical backpropagation, the intermediate output serves as the anchor point for calculating gradients across the distributed model parts. The label owner computes gradients with respect to the received intermediate outputs and sends these gradients back to the feature owners.

• Backward Pass Flow: The gradient of the loss relative to the intermediate output is propagated backwards to each feature owner's local model segment.
• Update Isolation: Each party uses these gradients to update only their portion of the split neural network, ensuring raw data never leaves its owner's device during the optimization process.

This concept is unique to the vertical data partition setting. It highlights a fundamental difference between Vertical (VFL) and Horizontal Federated Learning (HFL).

• VFL (This Context): Parties share different features on the same entities. They exchange intermediate outputs (activations) to complete a single forward/backward pass.
• HFL: Parties share the same feature space on different entities. They exchange model parameters (weights) or gradients after completing entire local forward/backward passes on their own data.

The role of the intermediate output extends beyond training into the vertical inference protocol for making predictions on new data. The production inference workflow mirrors the training forward pass.

• Prediction Flow: At inference time, a feature owner computes the intermediate output from a new sample and sends it to the label owner (or inference server).
• Result Generation: The label owner completes the forward pass through the remainder of the model to generate the final prediction (e.g., classification score). This enables collaborative prediction without exposing the query's raw features.

A model architecture where the neural network is divided into segments, with each segment residing on a different party in a Vertical Federated Learning (VFL) system. The division is defined by the cut layer. For example, a feature owner holds the initial layers processing its local features, while the label owner holds the subsequent layers leading to the final prediction. This structure enables distributed forward and backward propagation without sharing raw data.

The specific layer in a split neural network where the model is partitioned between parties. It determines the boundary of local computation:

• Feature owners compute the forward pass up to the cut layer, producing the intermediate output.
• This output is then securely transmitted to the next party (e.g., the label owner) to continue the forward propagation. The selection of the cut layer is a critical systems design choice, balancing privacy, communication cost, and model performance.

The distributed execution of a neural network's forward pass in VFL. It is a coordinated sequence:

1. Each feature owner computes the forward pass on its local features up to its designated cut layer.
2. Each produces an intermediate output (e.g., activations).
3. These intermediate outputs are securely aggregated or passed sequentially to the next party (often the label owner).
4. The final party completes the forward pass to compute the loss. This process enables collaborative inference without exposing the raw input features of any party.

The distributed computation of gradients in a split neural network during VFL training. After the vertical forward pass, gradients flow backwards:

• The label owner computes gradients for its portion of the model and the loss with respect to the received intermediate outputs.
• These gradients are then securely transmitted back to the respective feature owners.
• Each feature owner uses these received gradients to compute updates for its local model segment. This requires careful coordination to ensure gradient integrity across the partitioned model.

A cryptographic protocol for combining model updates (e.g., gradients or intermediate outputs) from multiple parties in VFL without revealing any individual contribution. This prevents a curious aggregator (like the label owner) from inferring sensitive information from a single party's update. Techniques often leverage Secure Multi-Party Computation (MPC) or homomorphic encryption to ensure that only the aggregated, masked result is revealed, providing a strong privacy guarantee during the collaborative learning process.

The defined sequence of communication and computation steps that coordinating parties follow to execute one complete training round in a VFL system. A standard protocol involves:

1. Entity Alignment (via PSI) to identify common samples.
2. Vertical Forward Propagation to compute the loss.
3. Vertical Backpropagation to calculate gradients.
4. Secure Aggregation of updates.
5. Model Update for each party's segment. The protocol must explicitly define roles, message formats, and security guarantees to ensure correct and private execution.