Ditto

Ditto introduces a proximal term to the local objective function that penalizes the deviation of local model parameters from the global model. This is formulated as:

• Local Objective: min_w [ L_k(w) + (λ/2) * ||w - w_global||² ]
• L_k(w) is the empirical risk on client k's local data
• λ (lambda) is a hyperparameter controlling the personalization-regularization trade-off
• Higher λ values enforce stronger adherence to the global consensus
• Lower λ values allow more aggressive local adaptation

This mathematical formulation provides a principled balance between global generalization and local specialization, preventing overfitting on small local datasets while still capturing site-specific patterns.

Ditto operates through a two-phase iterative process that decouples global aggregation from local personalization:

• Phase 1 - Global Update: The server aggregates client models using standard Federated Averaging (FedAvg) to produce a new global model w_global
• Phase 2 - Local Personalization: Each client solves its own regularized optimization problem independently, using the global model as an anchor point
• This decoupling means clients can run multiple local epochs without waiting for server synchronization
• The global model remains a stable reference point, preventing client drift in heterogeneous environments

This architecture is particularly effective when clients have non-IID data distributions with varying local sample sizes.

Ditto explicitly addresses statistical heterogeneity across federated clients by allowing each node to maintain a personalized model that diverges from the global consensus in a controlled manner:

• Handles label distribution skew where different hospitals have varying disease prevalence rates
• Accommodates feature distribution skew where patient demographics differ across geographic regions
• Manages quantity skew where some institutions have orders of magnitude more training samples than others
• The proximal term prevents clients with small datasets from overfitting while allowing data-rich clients to specialize
• Empirical results show Ditto outperforms both pure local training and standard FedAvg on heterogeneous benchmarks

Ditto maintains the same communication complexity as standard FedAvg while providing superior personalization:

• Only model weights are transmitted between server and clients, identical to FedAvg
• No additional communication rounds are required for the personalization mechanism
• The proximal term computation is entirely local and requires no extra data exchange
• Clients can perform multiple local update steps between communication rounds, reducing bandwidth requirements
• Compatible with gradient compression and quantization techniques for further communication reduction

This makes Ditto practical for deployment in bandwidth-constrained healthcare environments where frequent model synchronization is costly.

Ditto provides theoretical convergence guarantees under standard federated learning assumptions:

• Proven convergence rate of O(1/T) for strongly convex loss functions, matching FedAvg
• The proximal term ensures bounded client drift, a key factor in convergence analysis
• Convergence holds even when clients perform multiple local SGD steps between global synchronizations
• The framework is robust to partial client participation, where only a subset of nodes contribute in each round
• Theoretical analysis covers both i.i.d. and non-i.i.d. data distributions across clients

These formal guarantees provide confidence for deployment in safety-critical clinical applications where predictable model behavior is essential.

Ditto is designed as a plug-in personalization module that integrates with existing federated learning pipelines:

• Compatible with any neural network architecture including CNNs, RNNs, and transformers
• Works with standard optimizers: SGD, Adam, AdamW
• Can be combined with differential privacy mechanisms by adding noise to local updates
• Supports both cross-silo (hospital networks) and cross-device (wearable sensors) topologies
• The λ hyperparameter can be tuned per-client or set globally based on validation performance
• Open-source implementations available in popular FL frameworks including TensorFlow Federated and PySyft

A personalized variant of Federated Averaging that finds a shared initial model which clients can easily fine-tune with their local data. Unlike Ditto's proximal regularization approach, Per-FedAvg explicitly optimizes the Model-Agnostic Meta-Learning (MAML) objective, training for rapid adaptability rather than constrained local deviation.

• Uses second-order gradients to optimize for few-step personalization
• Computationally heavier than Ditto due to Hessian-vector products
• Ideal when clients need to adapt quickly to new local tasks

Partitions the neural network into a shared global representation and a personalized local head. While Ditto constrains the entire model via a proximal term, FedRep structurally separates what is shared from what is personalized.

• Base layers learn a common feature extractor across all clients
• Classification heads remain unique to each client
• Particularly effective when clients share feature representations but differ in label distributions
• Reduces communication overhead by only aggregating representation layers

Uses Moreau envelopes to decouple personalized model optimization from global model learning. This provides a smoother optimization landscape than Ditto's simple L2 proximal term, allowing each client to pursue a distinct local objective while maintaining a connection to the global consensus.

• Formulates personalization as a bilevel optimization problem
• More computationally intensive per-client than Ditto
• Provides stronger theoretical convergence guarantees under non-IID data

Takes a fundamentally different approach from Ditto: instead of personalizing a single global model, it recursively partitions clients into groups with similar data distributions. Each cluster maintains its own global model, effectively creating multiple concurrent personalization targets.

• Uses cosine similarity of model updates to identify clusters
• No explicit local objective modification like Ditto's proximal term
• Better suited when client populations form distinct, separable subgroups rather than a continuum of variation

A communication-efficient alternative to weight-based personalization. Rather than sharing model parameters with a proximal constraint like Ditto, clients share class scores or logits on a public dataset. A student model then learns from this heterogeneous teacher ensemble.

• Eliminates need for model architecture homogeneity
• Naturally handles statistical heterogeneity without explicit regularization
• Requires a representative public dataset, which Ditto does not
• Communication cost scales with output dimension, not model size

The simplest personalization baseline: take the globally aggregated model and perform additional training epochs on local data post-federation. Ditto formalizes this intuition by adding the proximal term during local training rather than as a separate post-processing step.

• No modification to the federated training loop required
• Risk of catastrophic forgetting without explicit regularization
• Ditto's proximal term provides a principled alternative to early stopping heuristics
• Often used as a comparison benchmark in personalization research