Federated Neural Architecture Search

The core search process for the optimal neural network topology is executed across distributed client devices without centralizing raw training data. Each client evaluates candidate architectures (e.g., via performance estimation strategies like one-shot NAS or gradient-based methods) on its local dataset. Only architecture-related metadata—such as performance metrics, gradients, or supernet weights—are shared with a central coordinator for aggregation. This preserves data privacy at the source while enabling global architectural discovery.

Federated NAS must minimize the communication overhead of searching vast architecture spaces. Key techniques include:

• Weight-sharing supernets: A single, over-parameterized network encapsulates all candidate sub-architectures. Clients train the shared supernet locally, and the server aggregates its weights, amortizing search cost.
• Performance predictors: Lightweight surrogate models learn to predict architecture accuracy from features, reducing the need for full local training of every candidate.
• Federated search space pruning: Early rounds identify and eliminate poorly performing architectural components globally, narrowing the search space communicated to clients.

Federated NAS must account for statistical heterogeneity (non-IID data) and system heterogeneity (varied device capabilities) across clients. Adaptations include:

• Personalized Architectures: The search can optimize for a family of architectures, where the final model is tailored to a client's data distribution or hardware constraints.
• Hardware-aware search objectives: The search reward function incorporates device-specific metrics like latency, memory footprint, or power consumption, ensuring discovered architectures are deployable.
• Robust aggregation: Techniques like client weighting based on data quantity or quality prevent the search from being biased by a subset of clients.

The search process itself must be designed to prevent leakage of private information from client data. This goes beyond secure model update aggregation and includes:

• Differential privacy for architecture gradients: Adding calibrated noise to the gradients of the supernet or architecture parameters before sharing.
• Secure multi-party computation (MPC) for ranking: Cryptographic protocols allow the server to compute the global ranking of candidate architectures without learning any client's individual performance metrics.
• Federated performance estimation: Techniques like federated knowledge distillation can be used to estimate architecture performance without transmitting raw logits or feature maps.

Federated NAS frequently leverages transferable architectural knowledge to accelerate search and improve results. Common patterns include:

• Warm-starting the supernet: Initializing the search supernet with weights pre-trained on a large, public source dataset (e.g., ImageNet).
• Search space design based on proven motifs: Constraining the search to include known efficient building blocks (e.g., inverted residuals, squeeze-and-excitation) transferred from prior NAS research.
• Cross-domain federated NAS: Searching for an architecture on a source task (e.g., image classification) and transferring the discovered topology to a related target task (e.g., object detection) across federated clients.

Federated NAS formalizes a bilevel optimization challenge within a distributed system. The outer loop searches for the optimal architecture α, while the inner loop optimizes the model weights w for that architecture on client data. The federated objective is: min_α E_{client i}[L_i(w_i*(α), α; D_i)] where w_i*(α) = argmin_w L_i(w, α; D_i) is the result of local training on client i's data D_i. Solving this requires coordinating architecture updates and weight updates across intermittent, heterogeneous clients, making convergence analysis and algorithm design uniquely complex.

FNAS enables hospitals to collaboratively search for the most accurate convolutional neural network (CNN) architecture for medical imaging tasks—like detecting tumors in MRI scans—without sharing sensitive patient data. Each institution trains candidate architectures locally, and only model performance metrics or gradients are shared for aggregation. This ensures compliance with regulations like HIPAA and GDPR while leveraging diverse clinical datasets to find robust, generalizable models.

Technology companies use FNAS to design efficient neural networks for features like next-word prediction or photo enhancement that run directly on users' devices. The search evaluates architectures based on real-world constraints like latency, memory footprint, and battery consumption across a heterogeneous fleet of phones. This results in personalized, performant models that respect user privacy by keeping all training data local.

Banks can employ FNAS to collaboratively develop superior anomaly detection models without exposing transaction details. The search explores architectures like graph neural networks (GNNs) or transformers capable of identifying complex, non-linear fraud patterns across decentralized transaction graphs. This collective intelligence improves detection rates for novel attack vectors while maintaining strict confidentiality of customer financial records.

Automakers use FNAS to evolve vision models for object detection and segmentation using data collected from vehicle fleets. The search process accounts for diverse geographic and weather conditions experienced by different cars. By transferring architectural knowledge from a source domain (e.g., simulation) and refining it federatedly on real-world edge data, companies accelerate the development of robust, safety-critical perception systems.

Neural Architecture Search is the automated process of discovering optimal neural network architectures for a given task and dataset. It treats the model's design—the connections, layers, and operations—as a searchable space.

• Core Components: A search space defines possible architectures, a search strategy (e.g., reinforcement learning, evolutionary algorithms) explores it, and a performance estimation strategy (e.g., training on a validation set) evaluates candidates.
• Goal: To automate and potentially surpass manual architecture engineering, finding high-performance models with constrained computational resources.

Federated Learning is a decentralized machine learning paradigm where a global model is trained collaboratively across multiple client devices (e.g., phones, sensors) holding local data samples, without exchanging the raw data itself.

• Process: Each client computes an update to the model based on its local data. Only these model updates (e.g., gradients or weights) are sent to a central server for secure aggregation into an improved global model.
• Primary Benefit: Enables model training on sensitive, distributed datasets while preserving data privacy and reducing communication costs compared to centralized data collection.

Transfer Learning is a technique where knowledge gained while solving one problem (the source task) is stored and applied to a different but related problem (the target task).

• Common Practice: Using a model pre-trained on a large, general dataset (e.g., ImageNet) as a starting point, then fine-tuning its parameters on a smaller, specific target dataset.
• Relevance to FedNAS: FedNAS often leverages transferable architectural patterns or warm-starts the search with architectures known to perform well on related source tasks, dramatically accelerating search convergence on the federated target task.

Differentiable Architecture Search is a gradient-based NAS method that formulates the search for discrete architectural components as a continuous optimization problem.

• Mechanism: It introduces a set of continuous architecture parameters that represent the strength of connections between operations. The search jointly optimizes these parameters and the model weights via standard gradient descent.
• Advantage for FedNAS: DARTS's efficiency makes it a candidate for federated settings, though it requires careful adaptation to handle non-IID data and communication of architecture parameters alongside model weights.

One-Shot Architecture Search is a family of NAS methods that train a single, over-parameterized supernet that encompasses all candidate architectures within the search space.

• Process: The supernet is trained once. Architectural candidates are then evaluated as different subnetworks of this supernet without needing separate training, enabling rapid performance estimation.
• Federated Application: In FedNAS, a supernet can be trained collaboratively across clients. The search then involves identifying the best-performing subnetwork for the aggregated data distribution, balancing global performance with potential client-specific personalization.

Multi-Task Learning is an approach where a single model is trained to perform multiple related tasks simultaneously, sharing representations between tasks to improve generalization and data efficiency.

• Principle: Inductive bias from training on auxiliary tasks helps the model learn more robust features that benefit the primary task.
• Connection to FedNAS: In a federated context, clients may have related but distinct tasks (e.g., different image classification categories). FedNAS can be extended to search for architectures that are inherently good at multi-task learning or personalization, creating a shared backbone with task-specific heads.