Federated Data Shard

The foundational principle of a data shard is that raw data never leaves the client's infrastructure. Only model updates—such as gradients or weights—are transmitted. This ensures compliance with data residency regulations like GDPR and HIPAA. The physical storage remains on the hospital's on-premise servers or a controlled cloud environment, maintaining complete local custody.

A shard is rarely a perfect microcosm of the global distribution. It typically exhibits non-IID (Non-Independently and Identically Distributed) properties due to local patient demographics or clinical practices.

• Label Distribution Skew: One hospital may have more positive cases of a rare disease.
• Feature Distribution Skew: Different imaging devices produce varying pixel intensity distributions.
• Concept Drift: The same label may have different local interpretations.

Each shard is managed by an autonomous entity, meaning the underlying database schema, file format, and data types are locally defined. A shard may consist of DICOM images, FHIR-compliant JSON records, or raw CSV lab results. Federated learning frameworks must handle this schema heterogeneity through a common data model mapping layer before local training can begin.

Before a shard can participate in training, it undergoes local Extract, Transform, Load (ETL) processes. This includes:

• De-identification: Stripping Protected Health Information (PHI).
• Normalization: Scaling pixel values or standardizing lab units.
• Tokenization: Converting clinical notes to token IDs. These operations execute entirely within the client's secure environment, preserving the privacy boundary.

A data shard is not static; it evolves as new patient records are added. It must be treated as a versioned asset with strict lineage tracking. Federated model registries often associate a specific model checkpoint with the exact hash of the shard version used for training. This provides a transparent audit trail for regulatory review and model reproducibility.

Unlike centralized data lakes, a shard is often processed on limited local hardware. Training may occur on a single GPU workstation or a small hospital cluster, not a hyperscale data center. This imposes constraints on the feasible model size and local batch size. Techniques like parameter-efficient fine-tuning (PEFT) are often required to make training viable on these resource-constrained shards.