Cross-Silo Federated Learning

Involves a small number of participants (e.g., 2-50 organizations like hospitals or banks), each acting as a reliable data silo. This contrasts with cross-device FL's millions of unreliable devices. The limited scale enables:

• Formal legal agreements (e.g., Data Use Agreements) governing participation.
• Higher resource guarantees per client (compute, memory, bandwidth).
• Simpler coordination and scheduling compared to massive, ephemeral device fleets.

Assumes a horizontal (feature-aligned) data split across silos. Each organization holds data for the same set of features but different entities (e.g., different banks hold customer records with identical feature columns). This is the most common and tractable partition for collaborative model training. Challenges arise from:

• Statistical heterogeneity (Non-IID): Data distributions vary significantly between silos (e.g., regional disease prevalence across hospitals).
• Varying data volumes: One hospital may have 10x more records than another, biasing the global model if not accounted for during aggregation.

The primary driver is regulatory compliance (HIPAA, GDPR) and institutional data sovereignty. Raw data never leaves its legal and physical jurisdiction. Privacy is enforced through a multi-layered stack:

• Algorithmic techniques: Differential privacy adds noise to model updates.
• Cryptographic protocols: Secure aggregation ensures the server only sees the sum of updates, not individual contributions.
• Trusted execution environments (TEEs): Hardware-enforced secure enclaves for local computation.
• Formal audits: Participants can audit the FL protocol and aggregation logic.

Operates over high-bandwidth, low-latency, and stable organizational networks (e.g., dedicated fiber, VPNs). This enables:

• Transmission of full model updates (e.g., 1GB model weights) per communication round.
• Synchronous training protocols where all selected clients participate simultaneously in each round.
• Reduced need for aggressive gradient compression, preserving model fidelity.
• Predictable training timelines, unlike cross-device FL which must handle frequent dropouts and unstable mobile connections.

Each client silo is a data center or server cluster with substantial computational resources (GPUs/TPUs). This allows:

• Multiple local epochs of training on the local dataset per communication round.
• Training of large, modern architectures (e.g., Vision Transformers, Large Language Models).
• Sophisticated local optimization (e.g., adaptive optimizers like Adam) without being bottlenecked by device compute.
• On-device validation and robust federated evaluation using held-out local test sets.

Participation is driven by clear mutual benefit, as the collaborative model outperforms any model trained on a single organization's data. This creates a cooperative game theory dynamic. Key considerations include:

• Incentive mechanisms: Ensuring fair contribution and benefit distribution (e.g., using Shapley values).
• Intellectual property (IP) management: Defining ownership of the final global model and any derived products.
• Fault tolerance for strategic dropouts: Protocols must handle a participant leaving the consortium without collapsing the training process.

The overarching decentralized machine learning paradigm where a global model is trained collaboratively across multiple edge devices or servers holding local data samples, without exchanging the raw data itself. Cross-Silo is a specific deployment scenario of FL characterized by a small number of reliable, resource-rich organizational participants.

• Core Principle: Keep training data localized; share only model updates (e.g., gradients, weights).
• Contrast with Cross-Device FL: Involves millions of unreliable mobile devices, whereas Cross-Silo involves tens to hundreds of stable organizational servers.

A rigorous mathematical framework for quantifying and limiting the privacy loss incurred by an individual when their data is included in a computation. In Cross-Silo FL, DP mechanisms are often applied during aggregation to provide a formal guarantee that the final model does not reveal whether any specific data point was in the training set.

• Key Mechanism: Adding calibrated statistical noise (e.g., Gaussian, Laplacian) to model updates or query outputs.
• Trade-off: Provides a quantifiable privacy budget (epsilon, δ) at the cost of some model utility or convergence speed.

A cryptographic protocol that allows a central server in a Federated Learning system to compute the sum of client model updates without being able to inspect any individual client's contribution. This is critical for Cross-Silo settings where silos (e.g., competing hospitals) require protection from the coordinator and from each other.

• How it works: Clients encrypt their updates using secret sharing or homomorphic encryption techniques. The server can only decrypt the aggregated sum.
• Benefit: Prevents inference attacks on individual model updates, complementing DP's protection of the final model.

Non-Independent and Identically Distributed (Non-IID) data is the statistical heterogeneity where data distributions vary significantly across different clients. This is the norm, not the exception, in Cross-Silo FL (e.g., one hospital specializes in cardiology, another in oncology).

• Primary Challenge: Causes client drift, where local models diverge, severely hindering global model convergence.
• Mitigation Strategies: Algorithms like FedProx (adds a proximal term to local loss) and personalization techniques are designed to handle this inherent data skew.

An advanced form of encryption that allows specific computations to be performed directly on ciphertext. In Cross-Silo FL, HE can enable the aggregation server to perform mathematical operations on encrypted model updates from clients, providing an even stronger privacy guarantee than Secure Aggregation alone.

• Use Case: The server receives encrypted gradients, adds them together while still encrypted, and sends the encrypted sum for decryption.
• Drawback: Computationally intensive, often making it impractical for large models but feasible for targeted, high-sensitivity aggregation in Cross-Silo.

An alternative distributed learning technique where a neural network is vertically partitioned between a client and a server. The client computes the initial layers and sends the intermediate activations (not raw data) to the server, which completes the forward and backward pass. This can be adapted for Cross-Silo scenarios.

• Contrast with FL: FL shares model parameters; Split Learning shares intermediate activations.
• Advantage in Cross-Silo: Can significantly reduce client-side compute and memory requirements, as the client only holds part of the model. Useful when silos have limited GPU resources but strong network connectivity.