Split Learning

The cut layer is the designated split point in the network architecture. The client retains the initial layers (feature extractor) and computes forward passes up to this layer. Only the smashed data (intermediate activations) is transmitted to the server, which holds the remaining layers. This ensures raw input data never leaves the client device. The choice of cut layer balances privacy (earlier cuts leak less semantic information) against compute offload (later cuts shift more work to the server).

Unlike Federated Learning's parallel client updates, Split Learning operates sequentially. The server trains with one client at a time in a lock-step fashion:

• Client computes forward pass to cut layer, sends activations
• Server completes forward pass, computes loss, initiates backprop
• Gradients flow back to cut layer, are sent to client
• Client completes backprop locally This sequential nature eliminates the need for client synchronization but introduces idle time for other clients.

In U-shaped Split Learning, labels remain on the client side. The server never sees the ground truth labels or the final loss computation. Instead, the server sends gradients back to the cut layer, and the client computes the final loss and updates its own layers. This provides stronger privacy guarantees than vanilla Split Learning, where the server holds the label head. U-shaped configurations are critical for applications where labels themselves are sensitive, such as medical diagnosis codes.

Split Learning enables resource-constrained clients (e.g., IoT sensors, mobile devices) to participate in training large models they could not host locally. By placing the cut layer early in the network, the client only executes a shallow feature extractor while the server handles the compute-intensive backbone. This is distinct from on-device training paradigms and makes Split Learning ideal for tinyML and edge scenarios where memory and FLOPs are severely limited.

SplitFed combines Split Learning's privacy properties with Federated Learning's parallel efficiency. Multiple clients perform split training concurrently with a central server, and the server-side model is aggregated via federated averaging. This hybrid approach:

• Reduces the idle time of pure sequential Split Learning
• Maintains raw data locality
• Enables parallel client participation SplitFed is particularly suited for cross-silo settings where institutions require both privacy and throughput.

Split Learning's communication occurs at every training iteration (forward activations and backward gradients), unlike Federated Learning which communicates only after multiple local epochs. This results in higher communication frequency but smaller payload sizes per transmission. The trade-off is sensitive to network latency: Split Learning performs well in low-latency, high-bandwidth environments but struggles with intermittent connectivity typical of cross-device FL scenarios.

The canonical configuration where a deep network is partitioned into a client-side bottom model and a server-side top model. The client computes forward passes up to a specific cut layer, transmits only the resulting smashed data (intermediate activations), and receives gradients for backpropagation. This prevents raw input exposure while keeping labels private on the server.

The cut layer is the designated split point in the neural network where computation transitions from client to server. The output at this boundary is called smashed data—a compressed, non-invertible representation of the raw input. Selecting the optimal cut layer balances:

• Privacy: Deeper cuts expose less raw information
• Compute: Shallow cuts offload more work to the server
• Bandwidth: The dimensionality of smashed data affects communication overhead

Unlike federated learning where labels reside with data owners, split learning keeps labels exclusively on the server. The server computes the loss (typically categorical cross-entropy) and backpropagates gradients through the top model. Only the gradient of the smashed data is returned to the client, ensuring the client never accesses ground-truth labels—a critical advantage for vertically partitioned collaborative scenarios.

Sequential split learning processes clients one at a time, with each client completing a full forward-backward pass before the next begins. This eliminates gradient staleness but introduces server idle time. Parallel split learning interleaves computation across multiple clients simultaneously, improving server utilization at the cost of managing asynchronous updates and potential weight staleness in the top model.

SplitFed combines split learning with federated averaging to mitigate the sequential bottleneck. Clients maintain local bottom models and a federated server aggregates client-side updates, while a separate split server manages the top model. This hybrid approach:

• Reduces client-side compute compared to pure federated learning
• Parallelizes training across clients
• Preserves label privacy on the split server

NoPeek is a defensive technique that minimizes the distance correlation between smashed data and raw inputs during training. By adding a regularization term to the loss function that penalizes statistical dependence, NoPeek ensures that intermediate activations reveal minimal information about the original data. This provides a quantifiable privacy guarantee without requiring differential privacy noise injection.