Federated Neural Architecture Search

Unlike centralized NAS, the search process is coordinated across multiple remote clients or edge devices, each holding private local data. A central server orchestrates the search by distributing a supernet or search policy, but never sees raw client data. Clients perform local architecture evaluations or updates and send only aggregated model updates or performance metrics back to the server. This orchestration requires robust protocols for client selection, synchronization, and handling of heterogeneous client capabilities and data distributions.

The primary driver is to perform NAS without centralizing sensitive training data, adhering to regulations like GDPR and HIPAA. Privacy is maintained through:

• Local Data Isolation: Raw data never leaves the client device.
• Secure Aggregation: Techniques like homomorphic encryption or secure multi-party computation can be applied to aggregated architecture gradients or scores.
• Differential Privacy: Adding calibrated noise to shared updates (e.g., architecture gradients or performance indicators) provides a mathematical privacy guarantee. This makes Federated NAS critical for domains like healthcare, finance, and on-device personalization.

Communication between clients and the server is the major bottleneck. Federated NAS methods optimize for this by:

• Weight Sharing in a Federated Supernet: A single, shared supernet is distributed. Clients train or evaluate sub-architectures locally, sharing only sub-network weights or gradients, not full model updates for every candidate.
• Federated Performance Prediction: Training a light-weight neural predictor on decentralized data to estimate architecture accuracy without full local training.
• Compressed Updates: Applying compression techniques like quantization or sparsification to the architecture-related gradients or parameters being communicated.

Client data is non-IID (not independently and identically distributed), meaning data distributions vary significantly between devices (e.g., different user writing styles on smartphones). This challenges NAS because:

• An architecture optimal for one client's data may perform poorly on another's.
• Local search progress can be biased and desynchronized. Solutions include personalized Federated NAS, which searches for architectures tailored to local data patterns, or using techniques like control variates to reduce the variance of architecture gradients aggregated from heterogeneous clients.

Federated NAS often targets deployment on diverse, constrained edge devices. The search naturally incorporates hardware-aware objectives measured directly on client hardware:

• Local Latency: Measuring inference time on actual client devices.
• Memory Footprint: Ensuring the architecture fits within device RAM.
• Energy Consumption: Optimizing for battery life. The search becomes a multi-objective optimization problem, balancing accuracy, latency, and privacy cost across a heterogeneous fleet, resulting in a Pareto frontier of architectures suited for different device classes.

Federated NAS is highly synergistic with Parameter-Efficient Fine-Tuning (PEFT) paradigms. Instead of searching full architectures, the search can be over efficient adaptation modules:

• Searching for optimal Adapter configurations, LoRA ranks, or prompt lengths for a frozen base model.
• Using a hypernetwork on the server to generate client-specific PEFT parameters based on aggregated signals. This drastically reduces the per-client communication and computation payload, as only small adapter weights or hypernetwork gradients are shared, making federated adaptation of large foundation models (e.g., LLMs) more feasible.

FNAS enables the collaborative development of specialized diagnostic models across multiple hospitals without sharing sensitive patient data. Institutions can jointly search for optimal architectures for tasks like medical image analysis (e.g., tumor detection in MRI scans) or clinical note processing. The resulting model benefits from diverse data distributions while maintaining strict compliance with regulations like HIPAA and GDPR. This avoids the legal and ethical risks of creating a centralized dataset.

This application focuses on designing efficient models for deployment on smartphones, IoT sensors, and other edge devices. FNAS searches for architectures optimized for constraints like low latency, small memory footprint, and energy efficiency using real data from the target devices. For example, a smartphone keyboard provider could use FNAS across millions of devices to discover the best next-word prediction model that balances accuracy with battery drain, all while keeping typing data private on the device.

Banks and financial institutions can use FNAS to build superior fraud detection models. Each bank holds proprietary transaction data that cannot be pooled due to competitive and regulatory reasons. FNAS allows them to collaboratively search for neural network architectures that best identify sophisticated, non-linear fraud patterns. The process improves model robustness against novel attack vectors seen across the consortium while ensuring that no bank's raw customer transaction data ever leaves its secure perimeter.

Beyond a single global model, FNAS can be used to search for personalized architectures or personalization layers suited to individual user data patterns. The search process evaluates which architectural components (e.g., specific adapter modules) work best for different user clusters within the federation. This leads to a system that can deploy a slightly different, optimally sized model to different users or devices, improving accuracy for all while maintaining the efficiency benefits of federated learning.

In manufacturing and industrial settings, different factories (or even different production lines within a factory) generate unique sensor telemetry data. This data is commercially sensitive and often too large to centralize. FNAS can be employed to discover the optimal neural architecture for predictive maintenance or quality control by searching across these isolated data silos. The resulting model captures failure modes and patterns from the entire federation, leading to more reliable and generalized predictions than a model trained on any single factory's data.

A specialized application combines FNAS with hypernetworks. Instead of searching for a monolithic architecture, the federated search optimizes a hypernetwork that generates parameter-efficient fine-tuning (PEFT) weights, such as LoRA matrices or adapter parameters. Clients train the shared hypernetwork on their local data. The output is a single hypernetwork capable of generating highly effective, task-specific PEFT modules for new clients or tasks, enabling efficient and private adaptation of large foundation models across an organization.

Neural Architecture Search (NAS) is the automated process of discovering high-performing neural network architectures for a given dataset and task. It treats the network design—choices like layer types, connections, and hyperparameters—as a search problem.

• Core Methods: Includes reinforcement learning-based controllers, evolutionary algorithms, and gradient-based approaches like Differentiable NAS (DNAS).
• Objective: Typically optimizes for a primary metric like validation accuracy, but often incorporates secondary goals like model size or latency.
• Key Challenge: The search process is computationally intensive, often requiring thousands of GPU hours, which led to the development of more efficient methods like weight-sharing in one-shot supernets.

Federated Learning (FL) is a decentralized machine learning paradigm where a global model is trained collaboratively across multiple remote devices or siloed data centers (clients) without exchanging raw data.

• Core Mechanism: Clients compute updates (e.g., gradients) on their local data and send only these updates to a central server for aggregation (e.g., via FedAvg).
• Primary Motivation: To preserve data privacy and reduce communication costs associated with centralizing massive datasets.
• Key Challenge: Managing statistical heterogeneity (non-IID data) across clients and ensuring efficient communication. FNAS extends this paradigm from weight updates to architecture discovery.

A hypernetwork is a neural network that generates the weights for a separate, larger main network. It provides a powerful mechanism for conditional, dynamic, or efficient parameterization.

• Role in PEFT/NAS: Hypernetworks can be used to generate the parameters for small adapter modules or to define the weights of architectures discovered during a search. In a federated context, a hypernetwork could generate client-specific architectures based on local data characteristics.
• Efficiency Benefit: Decouples the number of trainable parameters from the size of the main model, aligning with PEFT objectives.
• Example: A hypernetwork takes a client or task embedding as input and outputs the weights for a LoRA module or a small neural block.

Hardware-Aware NAS incorporates direct measurements or accurate proxies of hardware performance metrics into the search objective to find architectures optimized for specific deployment targets.

• Optimization Targets: Metrics include inference latency, memory footprint, energy consumption, and compute operations (FLOPs).
• Relevance to FNAS: In federated settings, clients have heterogeneous hardware (phones, sensors, edge servers). A federated search must therefore find architectures that are performant across this diverse hardware landscape, or even tailor sub-architectures to specific device classes.
• Methodology: Often uses lookup tables or on-device profiling to estimate latency during the search.

The Once-For-All (OFA) network is a training paradigm where a single supernet is trained to support a large, pre-defined set of sub-architectures (varying in depth, width, kernel size) without retraining.

• Efficiency for Deployment: After one training cycle, many efficient sub-networks can be extracted for different hardware constraints.
• Connection to FNAS: The OFA philosophy is highly compatible with federated search. A federated supernet could be trained across clients, and the search reduces to finding the optimal sub-network configuration for a given task or device profile using the shared supernet weights.
• Benefit: Dramatically reduces the computational cost of evaluating candidate architectures during the search phase.

Automated Machine Learning (AutoML) aims to automate the end-to-end process of applying machine learning to real-world problems. FNAS is a specialized subfield at the intersection of two AutoML pillars.

• Encompasses: Automated data preprocessing, feature engineering, model selection, hyperparameter optimization (HPO), and neural architecture search (NAS).
• FNAS as Composite AutoML: FNAS automates both the model design (NAS) and the training framework (Federated Learning). It represents a move towards fully automated, privacy-preserving model development pipelines.
• Goal: To reduce the need for extensive human ML expertise and manual tuning, especially in distributed, data-sensitive environments.