Horizontal Federated Learning

The defining architectural constraint of HFL is that all participating clients must share the same feature space (schema). This means every hospital, bank, or device holds data with identical column structures (e.g., age, blood pressure, glucose level) but for different patient populations. This alignment enables direct mathematical aggregation of local model parameters without requiring entity resolution or feature engineering across silos.

Data is partitioned horizontally across nodes based on sample IDs, not features. This is often described as 'row-based' partitioning. In contrast to Vertical Federated Learning, no single party holds a complete feature vector for any given sample. The collaborative goal is to increase the effective training dataset size by pooling statistically diverse samples while keeping each institution's records local.

Unlike idealized distributed machine learning in data centers, HFL clients almost never hold Independent and Identically Distributed (IID) data. A hospital in a rural area will have a drastically different patient demographic distribution than an urban clinic. This statistical heterogeneity causes client drift, where local optima diverge from the global optimum, making naive averaging algorithms like FedAvg unstable or slow to converge.

The central server must orchestrate rounds of synchronized updates with potentially thousands of geographically dispersed clients. The uplink speed from edge devices or institutional firewalls is often the primary constraint, not local compute. This necessitates communication-efficient protocols that minimize payload size through techniques like gradient compression, quantization, and sparsification to make training feasible over wide-area networks.

Because raw gradients can leak sensitive information through gradient leakage attacks, HFL systems must employ cryptographic defenses. The aggregation server must compute the weighted average of client updates without inspecting individual contributions. This is typically achieved using Secure Aggregation Protocols based on Secure Multi-Party Computation (SMPC) or Homomorphic Encryption, ensuring the server only sees the final aggregated model delta.

Not all clients are eligible to participate in every round. A client selection scheduler must balance statistical diversity against system constraints like battery life, connectivity, and compute availability. Furthermore, synchronous rounds are gated by the slowest participant. Straggler mitigation techniques—such as setting hard timeout deadlines, using asynchronous updates, or dropping late responders—are critical to prevent a single slow device from stalling the entire global training process.

The primary statistical challenge in Horizontal FL. Unlike centralized training where data is shuffled, local client datasets are not independently and identically distributed.

• Label Skew: One client may have mostly 'cat' images, another mostly 'dog' images.
• Feature Skew: Handwriting samples from different users have distinct stroke patterns.
• Quantity Skew: Massive variance in the volume of local data.
• Impact: Causes local models to diverge from the global optimum, slowing convergence or preventing it entirely.

The phenomenon where local model updates diverge from the global optimum due to heterogeneous Non-IID data. Each client's model overfits to its local distribution, pulling the aggregated global model in conflicting directions.

• FedProx Mitigation: Adds a proximal term to the local objective, penalizing large deviations from the global model.
• SCAFFOLD Mitigation: Uses control variates to correct for client drift during local updates.
• Symptom: High variance in validation accuracy across clients after aggregation.

Horizontal FL is deployed in two distinct topological contexts that dictate system design.

• Cross-Device: Massive scale (10^6+ clients), unreliable connectivity, limited compute. Used in Gboard and on-device personalization.
• Cross-Silo: Small number of institutional clients (2-100), reliable infrastructure, high compute. Used in hospital consortiums or bank collaborations.
• Key Difference: Cross-silo assumes stateful, available clients; cross-device must handle stragglers and intermittent participation.