Model Personalization

Personalization must balance global generalization (learned from the entire population) with local specialization (learned from a single client). Pure local training risks catastrophic forgetting of robust global features, while pure global models fail to capture individual nuances. Techniques like FedAvg with local fine-tuning or model interpolation (mixing global and local weights) are used to find the optimal Pareto frontier between these competing objectives.

Full model fine-tuning is computationally prohibitive on edge devices. Parameter-Efficient Fine-Tuning (PEFT) methods adapt large models by updating only a tiny fraction of parameters:

• Adapters: Small bottleneck layers inserted between frozen base model layers
• LoRA (Low-Rank Adaptation): Injects trainable rank-decomposition matrices into attention layers
• Prompt tuning: Learns soft prompts while keeping the base model frozen These techniques reduce on-device compute and communication costs by orders of magnitude.

A single user's data is inherently non-IID relative to the global population. Personalization addresses this client drift by allowing the model to shift its decision boundary toward the local distribution. This is critical for applications like next-word prediction on mobile keyboards, where vocabulary and phrasing are highly idiosyncratic. Without personalization, a generic model would consistently underperform for users with atypical language patterns.

The personalization process operates entirely on-device, ensuring raw data never leaves the client:

1. A global model is distributed to all clients via federated training
2. Each client fine-tunes the model on local data using on-device training
3. Only anonymized, aggregated model updates (gradients or deltas) are shared back
4. The server incorporates these updates to improve the next global model iteration This loop ensures that personalization benefits the collective without exposing individual records.

Federated meta-learning frameworks like Per-FedAvg train a global model specifically optimized for fast personalization. The objective shifts from finding a single model that performs well on average to finding an initialization from which a client can adapt to its local distribution in just a few gradient steps. This is analogous to teaching the model how to learn from limited personal data rather than memorizing population-wide patterns.

Model personalization can be framed as a multi-task learning problem where each client represents a distinct but related task. Techniques like MOCHA (MOdel-Agnostic meta-learning for Federated Personalization) exploit this structure by learning shared representations while allowing client-specific head layers. This approach is particularly effective when client data distributions are clustered—for example, users grouped by demographic or behavioral cohorts sharing latent similarities.

The process of performing local model optimization directly on edge hardware using locally stored data. This enables personalization by fine-tuning a global model on individual user interactions before sharing only abstracted updates with a federated server. Key characteristics include:

• Local SGD: Stochastic gradient descent computed entirely on-device
• Privacy by design: Raw data never leaves the device
• Resource constraints: Must operate within limited memory, battery, and compute budgets
• Intermittent connectivity: Training continues even when offline, with updates queued for later transmission

A set of adaptation methods that update only a small subset of model parameters or lightweight adapters, enabling efficient on-device personalization of large models in federated settings. Common techniques include:

• LoRA (Low-Rank Adaptation): Injects trainable rank decomposition matrices into frozen layers
• Adapter modules: Small bottleneck layers inserted between existing network components
• Prompt tuning: Learns soft prompts while keeping the base model frozen
• BitFit: Trains only bias terms, leaving weight matrices untouched

PEFT dramatically reduces communication overhead since only adapter weights need to be transmitted, not the full model.

A framework that combines meta-learning with federated optimization to train a global model that can rapidly adapt to new, unseen client distributions with only a few local gradient steps. The core mechanism:

• Model-Agnostic Meta-Learning (MAML) is adapted to federated settings
• The global model learns an initialization that is optimally positioned for fast personalization
• New clients can achieve high accuracy after just 1-5 local update steps
• Particularly effective when client data distributions are highly heterogeneous

This approach decouples the collaborative learning phase from the personalization phase, allowing each client to adapt independently.

The divergence of local model updates from the global optimum caused by heterogeneous, non-IID data distributions across clients. In personalization contexts, client drift presents both challenges and opportunities:

• Challenge: Excessive drift degrades the global model's generalization
• Opportunity: Controlled drift enables meaningful personalization
• Mitigation techniques: Proximal regularization (FedProx), variance reduction, and adaptive aggregation
• Personalization-aware aggregation: Some frameworks intentionally preserve beneficial drift components that capture local preferences

Understanding drift dynamics is essential for balancing global knowledge sharing against local adaptation quality.

A training paradigm where a federated model learns sequentially from a stream of non-stationary client data over time. This directly supports model personalization by:

• Adapting to evolving user preferences without full retraining
• Mitigating catastrophic forgetting through elastic weight consolidation or replay buffers
• Enabling lifelong personalization where models improve with extended use
• Handling concept drift where the underlying data distribution shifts seasonally or behaviorally

Continual learning ensures that personalized models remain relevant as user behavior changes, rather than degrading from stale initial training.

The algorithmic objective of ensuring that a collaboratively trained federated model performs equitably across diverse client populations. In personalization contexts, fairness considerations include:

• Performance parity: Personalized models must not systematically underperform for minority user groups
• Representation bias: Client selection strategies must include diverse data distributions
• Personalization vs. fairness trade-off: Aggressive personalization can amplify existing biases in local data
• Fair aggregation: Weighted averaging schemes that compensate for underrepresented clients

Effective personalization requires fairness-aware optimization to prevent the creation of discriminatory user experiences.