Vertical Communication Overhead

This is the core communication event in VFL. During the forward pass, each feature owner computes the activations up to the cut layer of their local split neural network segment. These intermediate outputs must be transmitted to the label owner (or the next party in the chain) to continue the forward propagation and compute the loss. The size of these outputs, which scales with the batch size and the dimensionality of the cut layer, directly determines a significant portion of the per-round bandwidth cost.

Following the backward pass, gradients must flow back through the split network. The label owner calculates gradients for its portion of the model and the immediate preceding layer. These gradients are then sent back to the corresponding feature owners, who use them to update their local model segments. This creates a bidirectional communication flow each round. For complex models, these gradients can be as large as the intermediate outputs, effectively doubling the core data transfer requirement per training iteration.

To ensure privacy, raw intermediate outputs or gradients are rarely sent in plaintext. The application of privacy-preserving techniques introduces substantial overhead:

• Homomorphic Encryption (HE): Allows computation on encrypted data but drastically increases the size of transmitted ciphertexts (by 10-100x) and the computational load for encryption/decryption.
• Secure Multi-Party Computation (MPC): Requires multiple rounds of communication between parties for a single operation, increasing latency.
• Differential Privacy (DP): Adding noise is computationally cheap, but coordinating noise scales and ensuring privacy budgets adds protocol complexity. This cryptographic layer is often the dominant factor in vertical computation overhead and latency.

Beyond the model data, VFL requires meta-communication to orchestrate the training process. This includes:

• Entity Alignment using protocols like Private Set Intersection (PSI), which involves multiple communication rounds to confidentially match sample IDs.
• Exchange of control messages to initiate rounds, signal readiness, and handle failures.
• Transmission of hyperparameters, model architecture updates, or convergence signals. While individually small, this traffic is essential for system operation and adds fixed latency, especially in wide-area network (WAN) environments with high round-trip times.

The effective overhead is magnified by disparities between participating parties. System heterogeneity leads to:

• Straggler Effect: Slower parties (due to weak hardware or busy networks) delay the entire synchronous training round, causing idle time for faster parties and reducing system efficiency.
• Intermittent Connectivity: Edge devices or cross-organizational servers may drop offline, requiring retransmissions or recovery protocols that consume additional bandwidth.
• Asymmetric Links: Upload speeds (from feature owners) are often much slower than download speeds, creating a bottleneck for sending intermediate outputs. Managing this heterogeneity is a key focus of vertical federated optimization.

A primary method to reduce overhead is to minimize the size of communicated messages. Key techniques include:

• Gradient/Activation Compression: Using lossy (e.g., quantization to lower precision like 8-bit integers) or lossless (e.g., entropy encoding) methods to shrink data size before transmission.
• Sparsification: Only sending the largest or most significant values (e.g., top-k gradients) and accumulating the rest locally. This can reduce communication volume by over 99% in some cases.
• Communication Frequency Reduction: Performing multiple local vertical gradient computation steps before communicating an update (similar to local steps in horizontal FL). These methods are central to communication-efficient federated learning but introduce a trade-off with model convergence speed and accuracy.

The design of the split neural network itself is a primary lever for reducing communication. Key strategies include:

• Strategic Cut Layer Placement: Choosing a cut layer early in the network (e.g., after the first few layers) drastically reduces the size of the intermediate outputs that must be transmitted compared to cutting at a deeper layer.
• Bottleneck Layers: Introducing dedicated compression layers (e.g., low-dimensional embeddings) at the cut point to create a smaller, information-dense representation for transmission.
• Feature Selection: Employing federated feature selection techniques to identify and use only the most informative features from each party, reducing the input dimension and thus the subsequent computation and communication.

Instead of sending full, dense gradients or intermediate results, these techniques transmit only the most critical information.

• Top-k Sparsification: Only the k largest magnitude values (gradients or activations) are transmitted; the rest are set to zero. The receiving party often uses error accumulation to maintain convergence.
• Quantization: Reducing the numerical precision of transmitted values (e.g., from 32-bit floats to 8-bit integers). Stochastic quantization can provide unbiased estimates with lower bandwidth.
• Gradient Clipping: Bounding gradient values to a fixed range not only aids stability but also makes subsequent compression and quantization more effective and predictable.

Reducing how often parties need to communicate, not just the size of each message.

• Local Steps (Vertical Federated Optimization): Allowing feature owners to perform multiple local forward/backward passes on their model segment, accumulating updates before communicating with the label owner. This turns many communication rounds into one.
• Lazy Aggregation: The server (label owner) does not request updates from all parties every round. Updates are only pulled when they are estimated to be sufficiently informative, based on heuristics or validation performance.
• Asynchronous Protocols: Moving away from synchronous lock-step training, where the system waits for the slowest party. Faster parties can proceed with more frequent updates, improving hardware utilization and reducing idle time.

The cryptographic protocols used for privacy introduce significant overhead. Optimizing them is crucial.

• Efficient Homomorphic Encryption (HE) Schemes: Choosing partial homomorphic encryption (e.g., Paillier) over fully homomorphic encryption when the required operations (typically addition) are limited. Newer schemes like CKKS allow approximate arithmetic on encrypted vectors, balancing precision and speed.
• Lightweight Secure Aggregation: Designing vertical secure aggregation protocols that minimize the number of cryptographic rounds and the size of masks or keys exchanged. Some protocols leverage efficient secret sharing instead of heavier HE for aggregation.
• Hybrid Privacy Approaches: Combining lightweight differential privacy (adding noise) with selective use of cryptography. For less sensitive intermediate data, DP may suffice, avoiding encryption costs entirely.

Avoiding redundant computation and transmission by storing and reusing previously computed values.

• Activation Caching: During training, the intermediate outputs for a given batch of aligned entities can be cached by the feature owner. If the local model segment hasn't changed significantly (e.g., after a small local update), these cached values can be reused for several global rounds, skipping forward passes.
• Lookup Table (LUT) for Non-Linearities: For fixed non-linear activation functions (e.g., ReLU, sigmoid), pre-computed encrypted lookup tables can be used to evaluate the function on encrypted data more efficiently than a full cryptographic evaluation each time.

Leveraging software infrastructure to manage communication intelligently.

• Vertical FL Framework Optimizations: Modern vertical FL frameworks implement batching, pipelining, and efficient serialization (e.g., Protocol Buffers, FlatBuffers) to reduce protocol overhead and latency.
• Network-Aware Scheduling: The orchestrator considers current network bandwidth and latency between parties when scheduling updates, prioritizing communication between well-connected nodes or during off-peak times.
• Compression-Accelerated Hardware: Utilizing hardware (e.g., GPUs with tensor cores) that natively supports low-precision computation and network cards that support in-line compression to reduce the effective data transferred.

The defined sequence of communication and computation steps that coordinating parties follow to execute a complete training round in a VFL system. This protocol dictates the frequency, direction, and content of all messages exchanged, directly determining the communication overhead. A poorly designed protocol can lead to excessive rounds of encrypted data exchange.

• Example: A protocol may specify that in each round, feature owners compute intermediate outputs, encrypt them, send them to the label owner, receive back encrypted gradients, and then update their local model segment.

The model architecture used in VFL where the neural network is divided into multiple parts, with each part residing on a different party. The cut layer defines the division point. This split necessitates the exchange of intermediate outputs and gradients between parties during every forward and backward pass, which is the primary source of communication overhead. The location of the cut layer is a major engineering trade-off between model performance and communication cost.

The result of the forward pass up to the cut layer, computed by a feature owner on its local features. This tensor must be securely transmitted to the next party (typically the label owner) to continue the forward propagation. The size and dimensionality of this output, which depends on the cut layer's design, are direct drivers of per-round communication bandwidth. For a layer with 256 neurons, transmitting this output for a batch of 32 samples creates significant data transfer.

Cryptographic protocols used to combine model updates (e.g., gradients) from multiple feature owners without revealing any individual party's contribution. While essential for privacy, these protocols introduce significant communication and computation overhead. Techniques like Secure Multi-Party Computation (MPC) or masking schemes require multiple rounds of communication between parties to establish secrets and perform aggregation, adding latency and bandwidth costs beyond the raw data transfer.

The application of encryption schemes that allow computations (e.g., addition, multiplication) to be performed directly on ciphertext. In VFL, this allows the label owner to compute on encrypted intermediate outputs from feature owners. While it provides strong privacy, Homomorphic Encryption drastically increases communication overhead because ciphertexts are orders of magnitude larger than plaintexts. It also adds substantial computation overhead for encryption and decryption operations on all exchanged data.

The additional local processing cost incurred by each party in a VFL system beyond standard neural network operations. This is a companion cost to communication overhead and includes:

• Cryptographic Operations: Encrypting/decrypting intermediate outputs and gradients.
• Partial Model Execution: Running only a segment of the full model.
• Protocol Logic: Managing secure aggregation and multi-party computation steps.

High computation overhead can bottleneck the system, slowing training rounds and increasing effective latency.