Vertical Backpropagation

Unlike standard backpropagation, gradients are computed in a coordinated, distributed manner across the participating parties. The label owner calculates gradients for its segment of the split neural network and sends specific, often encrypted, gradient signals backwards to each feature owner. Each party then uses these signals to compute updates exclusively for the model parameters attached to their locally held features.

The core challenge is transmitting gradient information without leaking sensitive feature data. This is addressed via privacy-preserving techniques:

• Homomorphic Encryption: Allows the label owner to compute on encrypted intermediate outputs from feature owners and send back encrypted gradients.
• Secure Multi-Party Computation (MPC): Protocols that enable joint gradient calculation where no party sees another's raw input.
• Gradient Masking: Adding noise or applying transformations to gradients before sharing to obfuscate the underlying data.

The cut layer—where the model is split between parties—dictates the mechanics of backpropagation. The gradient calculated at the cut layer becomes the critical signal passed backward:

• A shallow cut (early in the network) means feature owners perform little local computation but must share more detailed gradient information.
• A deep cut (later in the network) increases local computation for feature owners but reduces the complexity and potential privacy leakage of the exchanged gradients. The placement is a trade-off between privacy, communication cost, and computational load.

Parties have heterogeneous responsibilities during backpropagation:

• Label Owner: Acts as the coordinator. It computes loss, initiates backpropagation for its network segment, and manages the secure aggregation or distribution of gradients to feature owners. It often bears the highest computational burden.
• Feature Owners: Compute gradients for their local model components based on the received signals. Their workload is typically lighter but includes potential cryptographic overhead. They never see the raw loss or gradients from other feature owners.

Vertical backpropagation introduces significant communication overhead not present in centralized training. Each training iteration requires multiple synchronous communication rounds:

1. Forward pass: Intermediate outputs sent to label owner.
2. Backward pass: Gradients sent back to feature owners. This round-trip latency, combined with the bandwidth needed for encrypted messages, often becomes the primary bottleneck, limiting model complexity and training speed.

Execution requires a strict, predefined vertical training protocol. This protocol sequences all operations to ensure correctness and privacy:

• Synchronization Steps: Ensures all parties are at the correct stage (forward pass, gradient receipt, parameter update).
• Error Handling: Manages party dropouts or network failures mid-backpropagation.
• Aggregation Rules: Defines how the label owner combines gradient signals if multiple feature owners are involved. This structured coordination is essential for reproducible and secure model convergence.

Banks, credit card networks, and e-commerce platforms collaborate to train a fraud detection model. Each holds different features for the same customers (e.g., bank holds transaction history, e-commerce holds purchase patterns, credit bureau holds credit score). Vertical backpropagation allows the joint model to learn from this combined feature space. The label (fraudulent/legitimate transaction) is typically held by the bank. During training, gradients are computed across the split network to update each institution's model segment, creating a more robust detector without any party exposing its raw customer data.

Hospitals, clinical labs, and insurance providers collaborate to improve diagnostic models for diseases like cancer or diabetes. Data is vertically partitioned: the hospital holds medical imaging and doctor's notes, the lab holds genomic and blood test results, and the insurer holds longitudinal treatment and outcome data. Vertical backpropagation facilitates training a unified model. The lab or hospital may act as the label owner (diagnosis). Gradients for each segment are computed securely, allowing the model to learn from a complete medical profile while complying with strict regulations like HIPAA and GDPR, as patient records never leave their original institution.

A retailer partners with a social media platform and a payment processor to build a hyper-personalized recommendation engine. Features are split: retailer has purchase history and browsing data, social media has user interests and engagement, payment processor has spending categories and frequency. Vertical backpropagation enables the training of a joint model where the retailer holds the label (purchase/no-purchase). The algorithm computes gradients through the distributed network, optimizing each party's model component to predict user preference based on the fused feature context, significantly improving recommendation relevance without pooling sensitive user profiles.

An advertiser collaborates with multiple media publishers (e.g., a news site, a video platform, a search engine) to optimize click-through rate (CTR) prediction. Each publisher holds different behavioral features (article reads, video watches, search queries) for overlapping but anonymized user IDs. Vertical backpropagation is used to train the CTR model. The advertiser, who observes the final conversion label (click), orchestrates the backward pass. Gradients are propagated securely back through each publisher's model segment, allowing the system to learn which combined cross-platform signals best predict engagement, all while maintaining user privacy and platform data sovereignty.

In a smart city, different municipal departments hold vertically partitioned sensor data about the same urban infrastructure. The transportation department has traffic camera feeds, the utilities department has power grid load data, and the environmental agency has air quality sensor readings. They collaborate to build a model for predicting traffic congestion or public safety incidents. Vertical backpropagation allows gradients to flow through a model split across these departments. The department with the target event label (e.g., traffic control center for congestion) coordinates the training, enabling a holistic AI model that reasons across disparate data silos to improve city operations.

Multiple financial institutions, such as a traditional bank, a fintech lender, and a telecom provider, collaborate to assess borrower creditworthiness for customers who interact with all of them. Each holds a unique feature vertical: bank (account history, income), fintech (alternative lending history, app usage), telecom (bill payment history, plan type). Vertical backpropagation enables training a unified credit scoring model. The bank, which holds the definitive label (loan default history), computes loss and initiates the backward pass. Gradients are distributed to update each institution's component, creating a more accurate risk model that leverages a complete financial picture without any party disclosing proprietary underwriting data.

Vertical Federated Learning (VFL) is a decentralized machine learning paradigm where different parties (clients) hold different features (columns) about the same set of entities (rows). They collaborate to train a unified model without directly sharing their raw, vertically partitioned feature data. This is distinct from horizontal FL, where parties have different samples but the same features.

• Primary Use Case: Collaborative modeling between organizations with complementary data on shared customers (e.g., a bank with credit history and a retailer with purchase history).
• Core Challenge: Requires entity alignment to identify overlapping samples without exposing non-overlapping data.

A split neural network is the model architecture used in VFL where the network is divided into multiple segments, each residing on a different participating party. The division point is defined by the cut layer.

• Bottom Models: Held by feature owners. These are the initial layers that process each party's local feature subset.
• Top Model: Typically held by the label owner. This segment receives the concatenated intermediate outputs from all bottom models to produce the final prediction.
• Forward Pass: Each feature owner computes its local forward pass up to the cut layer, producing an intermediate output (embedding) that is sent to the label owner.

The cut layer is the specific layer in a split neural network where the model is partitioned between parties in a VFL system. It is a critical hyperparameter that determines the boundary between local and shared computation.

• Trade-off Governed by Cut Layer:
  • Privacy: A shallower cut (early split) keeps more of the data transformation local, enhancing privacy but potentially limiting model capacity.
  • Communication & Utility: A deeper cut (later split) allows for more complex joint representation learning but increases the size of the transmitted intermediate outputs and potential privacy leakage.
• Selection: The optimal cut layer is chosen based on the privacy budget, network bandwidth, and the desired model performance.

Vertical forward propagation is the distributed execution of the forward pass in a VFL system. It is the prerequisite step that generates the inputs for vertical backpropagation.

• Process Flow:
  1. Each feature owner computes the forward pass on its local model segment using its private features.
  2. Each produces an intermediate output (a vector or embedding) at the cut layer.
  3. These intermediate outputs are securely sent to the label owner.
  4. The label owner concatenates these inputs and completes the forward pass through its top model to compute the final prediction and loss.
• Security Consideration: The intermediate outputs must be protected (e.g., via encryption) as they can leak information about the raw input features.

Vertical secure aggregation encompasses cryptographic protocols used to combine updates from multiple parties in VFL without revealing any individual contribution. While often associated with gradient aggregation, it also applies to protecting intermediate outputs and gradients during backpropagation.

• Purpose: Prevents a curious label owner or other feature owners from inferring a specific party's private data from their shared model updates.
• Common Techniques:
  • Secure Multi-Party Computation (MPC): Parties jointly compute functions (like summation) over their private values.
  • Homomorphic Encryption (HE): Allows the label owner to perform computations on encrypted intermediate outputs or gradients.
• Impact on Backpropagation: These protocols add computation overhead and communication overhead to the gradient exchange steps.

A vertical training protocol is the complete, defined sequence of communication and computation steps that all parties follow to execute one training round in a VFL system. Vertical backpropagation is a phase within this protocol.

• Standard Protocol Steps:
  1. Entity Alignment: Securely identify common samples using Private Set Intersection (PSI).
  2. Vertical Forward Propagation: Compute intermediate outputs.
  3. Loss Calculation: Performed by the label owner.
  4. Vertical Backpropagation: Compute and distribute gradients.
  5. Model Update: Each party updates its local model segment using its received gradients.
• Orchestration: This protocol is typically implemented and managed by a Vertical FL framework, which handles coordination, failure recovery, and security enforcement.