Federated Feature Selection

Feature importance is assessed locally on each party's data subset. Common metrics like mutual information, chi-squared tests, or model-based importance scores (e.g., from a local Gini impurity or gradient analysis) are computed in isolation. The results are then aggregated via a secure protocol to form a global ranking, ensuring no raw feature values are exposed during evaluation.

Local feature importance scores or rankings must be combined without revealing any single party's contribution. This is achieved through cryptographic techniques:

• Secure Multi-Party Computation (MPC) for computing aggregate statistics.
• Homomorphic Encryption to allow computations on encrypted scores.
• Differential Privacy by adding calibrated noise to scores before aggregation, providing a mathematical privacy guarantee.

The process operates on a vertically partitioned dataset, where different parties hold different features for the same entities (e.g., customers). Therefore, federated feature selection is fundamentally preceded by entity alignment—securely identifying the overlapping set of samples across parties using protocols like Private Set Intersection (PSI). Selection is only meaningful for this aligned cohort.

Protocols are designed to minimize the bandwidth and rounds of communication between the label owner (who holds the target variable) and feature owners. Instead of sharing data, parties exchange compact representations:

• Encrypted summary statistics (e.g., covariance matrices).
• Intermediate outputs from a collaboratively computed model.
• Final feature scores or binary selection masks.

Selection methods vary based on the intended model:

• Model-Agnostic: Uses filter methods (e.g., federated correlation analysis) independent of the final ML algorithm.
• Model-Specific: Uses embedded methods where selection is part of the federated training. For example, in a split neural network, gradients or activation patterns at the cut layer can inform which input features are most influential.

Federated feature selection is not always a standalone pre-processing step. It can be interleaved with the vertical training protocol itself. Techniques like federated LASSO (using L1 regularization) or attention mechanisms within the federated model can perform continuous feature selection during training, dynamically down-weighting irrelevant features.

Before feature selection can begin, parties must identify their overlapping samples (entities) without revealing their full datasets. This requires Private Set Intersection (PSI) protocols, which are computationally intensive and can become a bottleneck, especially with large ID spaces. The communication and cryptographic overhead scales with the number of participating parties and the size of their datasets, often dominating the initial setup time.

Most feature selection algorithms rely on global dataset statistics (e.g., correlation matrices, mutual information, variance). Computing these in a federated setting requires multiple rounds of secure aggregation. For vertically partitioned data, calculating the correlation between a feature on Party A and a feature on Party B necessitates exchanging encrypted intermediate results, leading to an O(n²) communication complexity in the number of features, which can be prohibitive.

The final set of selected features itself can be a significant privacy leak. Revealing that a specific subset of features (e.g., 'genetic marker X' and 'purchase of medication Y') is highly predictive for a target label allows adversaries to infer sensitive business logic or patient conditions. Mitigation requires techniques like differential privacy on the selection criteria or performing selection on a trusted execution environment, adding another layer of complexity.

Different parties will have varying data quality, missing value patterns, and encoding schemas for categorical features. A federated feature selection algorithm must be robust to this heterogeneity. For example, a feature may appear highly informative from one party's clean data but be noisy from another's. This requires robust federated normalization and imputation strategies before selection can be reliably performed.

The choice of selection method is constrained by the federated architecture:

• Model-agnostic methods (e.g., filter methods like federated mutual information) are more flexible but may not align with the final model's inductive bias.
• Model-specific methods (e.g., wrapper methods using the federated model's gradients) are more accurate but require full, iterative training rounds for each candidate feature subset, dramatically increasing computational and communication costs.

Parties may have misaligned incentives. A feature owner whose data contributes little to the model may still incur computation/communication costs. Conversely, a party with highly predictive features may be reluctant to participate without compensation, leading to free-riding or data dropout. Designing incentive mechanisms and verifiable contribution scoring for feature selection is an open research and practical challenge.

A vertical data partition is a dataset split where different features (columns) describing the same set of entities (rows) are held by separate organizations. This is the foundational data structure for VFL.

• Example: Hospital A holds patient lab results, Hospital B holds imaging data, and an insurer holds diagnosis labels—all for the same overlapping patient population.
• The core challenge is to train a model using all features without centralizing the columns, which is the problem federated feature selection aims to solve.

Entity alignment is the prerequisite process of securely identifying the overlapping records (e.g., the same customers or devices) across the vertically partitioned datasets held by different parties.

• Without alignment, parties cannot know which of their features correspond to the same sample for training.
• This is typically achieved using privacy-preserving protocols like Private Set Intersection (PSI) to compute the common set of entity IDs without revealing non-overlapping records.

Private Set Intersection (PSI) is a cryptographic protocol that allows two or more parties, each holding a private set of items (e.g., user IDs), to compute the intersection of their sets without revealing any information about items not in the intersection.

• Critical Role in VFL: PSI is used for the initial entity alignment step, enabling parties to discover their common samples to train on.
• It ensures that the raw ID lists remain confidential, preserving privacy before feature-based collaboration even begins.

A split neural network is the standard model architecture for VFL, where the neural network is divided into multiple segments hosted by different parties.

• Feature Owners hold the bottom layers that process their local features.
• The Label Owner (or a coordinator) holds the top layers closer to the output.
• The Cut Layer is the boundary where intermediate outputs (embeddings) are shared from feature owners to the label owner to complete the forward pass. Federated feature selection often determines which features feed into this split architecture.

Homomorphic Encryption (HE) is a cryptographic technique that allows computations to be performed directly on encrypted data. In VFL, it's used to protect the intermediate results exchanged during training.

• A feature owner can encrypt their computed embeddings before sending them to the label owner.
• The label owner can perform specific operations (e.g., aggregation) on these ciphertexts to compute gradients.
• This prevents the label owner from directly inspecting the raw feature representations, adding a strong layer of privacy during federated feature evaluation and training.

Vertical secure aggregation refers to protocols that securely combine model updates (e.g., gradients or feature importance scores) from multiple feature owners without revealing any single owner's contribution.

• This is distinct from horizontal FL's secure aggregation; here, updates are from different parts of the model (different features).
• It is essential for privacy in federated feature selection, as it prevents the label owner from inferring which features from a specific party were most influential.
• Techniques often leverage Multi-Party Computation (MPC) or Homomorphic Encryption.