Vertical Federated Optimization

In VFL, the feature distribution across parties is inherently heterogeneous. One party may hold highly predictive features while another holds noisy or weakly correlated ones. This creates an imbalanced contribution to the global model's learning, leading to convergence instability and requiring specialized optimization techniques that account for statistical skew across the vertical partition.

Each training iteration requires the exchange of intermediate outputs and gradients between the label owner and feature owners. This creates a significant vertical communication overhead. The primary challenges are:

• Bandwidth Cost: Transmitting high-dimensional embeddings or gradients for every aligned sample.
• Synchronization Latency: The slowest participating party dictates the pace of each training round.
• Protocol Complexity: Secure aggregation and encryption add further communication rounds. Optimization must minimize the frequency and volume of these exchanges.

Privacy guarantees in VFL often rely on homomorphic encryption or secure multi-party computation (MPC). These cryptographic primitives impose a heavy vertical computation overhead:

• Encryption/Decryption Latency: Operations on ciphertext are orders of magnitude slower than plaintext computations.
• Ciphertext Expansion: Encrypted data can be 100-1000x larger, exacerbating communication costs.
• Secure Protocol Rounds: Each mathematical operation may require multiple communication rounds between parties. Optimization algorithms must be designed to be crypto-friendly, minimizing the number of multiplicative operations and interactive rounds.

Unlike horizontal FL, a feature owner in VFL only computes gradients for its segment of the split model. This leads to partial gradient updates. Key issues include:

• Vanishing/Exploding Gradients: Can be amplified when passed through the cut layer.
• Update Staleness: Feature owners may update at different frequencies due to system heterogeneity.
• Coordinated Optimization: Standard optimizers like Adam or SGD assume access to the full gradient vector, which is not available to any single party in VFL. This necessitates federated optimizer designs that can operate on fragmented state information.

VFL systems must operate across organizations with vastly different infrastructure profiles. This heterogeneity manifests in:

• Compute Disparity: Parties may have different GPU/CPU capabilities, causing stragglers.
• Availability Windows: Participants may only be online intermittently due to internal policies.
• Data Schema Drift: The statistical properties of a party's feature columns can change independently over time. Optimization protocols must be robust to dropout and capable of handling dynamic feature spaces without catastrophic failure.

Proving that a VFL optimization algorithm will converge is complex due to the distributed, encrypted, and non-IID nature of the problem. Key theoretical challenges include:

• Analyzing Split Architectures: Convergence rates depend on the choice of cut layer and the sensitivity of the loss function to each party's features.
• Accounting for Noise: Privacy mechanisms like differential privacy add noise, which must be incorporated into the convergence analysis.
• Bounded Coordination Assumptions: Proofs often rely on assumptions about synchronization and honest participation that may not hold in practice. Developing theoretically sound VFL optimizers with proven convergence under realistic conditions is an active research frontier.

A split neural network is the core architectural pattern for Vertical Federated Learning (VFL), where a single model is divided into multiple segments hosted by different data owners. The division occurs at a designated cut layer. Feature owners compute the forward pass up to this layer, producing an intermediate output (e.g., embeddings or activations), which is securely sent to the label owner. The label owner completes the forward pass, calculates the loss, and initiates vertical backpropagation to distribute gradients back to each segment for local updates. This design ensures no party ever has access to the complete model or the full raw feature set of any entity.

Private Set Intersection (PSI) is a foundational cryptographic protocol for entity alignment in VFL. It allows two or more parties, each holding a private set of identifiers (e.g., customer IDs), to jointly compute the intersection of their sets—discovering which entities they have in common—without revealing any information about identifiers not in the intersection. For example, a bank and a retailer can find their shared customers without exposing their full client lists. PSI is critical for establishing the aligned sample space required for training without compromising data privacy during the initial setup phase of a VFL system.

Secure Multi-Party Computation (MPC) provides the cryptographic foundation for privacy-preserving computations in VFL. MPC protocols enable a group of distrusting parties to jointly compute a function (e.g., a gradient or a loss value) over their private inputs while revealing only the final output. In the context of vertical multi-party computation, this allows feature and label owners to collaboratively train a model. Common techniques include:

• Secret Sharing: Splitting data into random shares distributed among parties.
• Garbled Circuits: Encoding logic gates for secure evaluation. These protocols prevent any single party from reconstructing another's raw data during training or inference.

Homomorphic Encryption (HE) is an encryption scheme that allows computations to be performed directly on ciphertext. In VFL, a feature owner can encrypt their intermediate output before sending it to the label owner. The label owner can then perform specific operations (e.g., addition, multiplication) on this encrypted data during the forward and backward passes. The result, when decrypted by the feature owner, matches the result of operations performed on the plaintext. This enables training and inference where the label owner never sees the feature owners' data in a decrypted form, providing a strong layer of cryptographic privacy. Partially Homomorphic Encryption (PHE), which supports only one operation type, is often used for efficiency.

Vertical Federated Averaging (VFA) is a family of collaborative optimization algorithms adapted from the classic FedAvg algorithm for the horizontally partitioned setting. In VFL, 'averaging' does not refer to model weights, as each party holds a different part of the model. Instead, VFA mechanisms coordinate the aggregation of updates (e.g., gradients or model segment parameters) from feature owners. The process involves:

• Synchronizing training rounds across aligned entities.
• Securely transmitting and aggregating gradient contributions.
• Updating each party's local model segment with the aggregated result. These algorithms must account for system heterogeneity and the statistical challenges of vertically partitioned features to ensure stable convergence.

Differential Privacy (DP) provides mathematical, worst-case privacy guarantees by ensuring a model's output does not significantly change with the inclusion or exclusion of any single individual's data. In Vertical FL with Differential Privacy, DP mechanisms are applied to prevent leakage from shared information like gradients or intermediate outputs. Common techniques include:

• Gradient Perturbation: Adding calibrated noise (e.g., Gaussian) to gradients before they are shared.
• Output Perturbation: Adding noise to the intermediate embeddings sent from feature owners to the label owner. This creates a formal privacy budget (epsilon, δ) that quantifies the privacy-utility trade-off, allowing organizations to comply with regulations like GDPR while collaboratively training models.