On-Device Evaluation

Inference latency is the time elapsed between submitting an input to a model and receiving its prediction, measured in milliseconds (ms) on the target device. This is the primary determinant of user-perceived responsiveness.

• Real-time vs. Batch: Critical for interactive applications (e.g., voice assistants, AR) versus offline processing.
• Factors: Heavily influenced by model architecture, hardware accelerator (CPU/GPU/NPU), and input data size.
• Benchmarking: Must be measured under realistic thermal and power constraints, not just peak performance.

Model accuracy is the proportion of correct predictions made by the model on a local dataset. The F1 score (the harmonic mean of precision and recall) is preferred for imbalanced class distributions common in on-device data.

• Local vs. Global Discrepancy: A key federated learning challenge; local accuracy can diverge significantly from global test set accuracy due to non-IID data.
• Personalization Signal: Tracking local accuracy over time is essential for evaluating personalized federated learning techniques.
• Evaluation Set: Requires a held-out portion of the device's private data that was not used for local training or fine-tuning.

Energy consumption quantifies the total electrical energy used by the device to perform model inference or training, measured in Joules or milliwatt-hours (mWh). This is a paramount constraint for battery-powered IoT and mobile devices.

• Power Profiling: Measured using hardware monitors or estimated via performance counters (e.g., CPU cycles, memory accesses).
• Trade-offs: Directly conflicts with latency and accuracy; simpler models consume less energy.
• System Impact: Must account for the energy cost of sensor data pre-processing and result post-processing, not just the neural network inference.

Memory footprint encompasses the total RAM (working memory) and storage (persistent memory) required to load and execute the model, including weights, activations, and intermediate buffers.

• Peak RAM Usage: The maximum working memory required during inference, which must fit within the device's available RAM.
• Model Size: The size of the quantized and serialized model file on disk or flash storage. Techniques like post-training quantization and pruning directly reduce this.
• Constraint: For microcontrollers (TinyML), models often must fit in < 1MB of total flash and RAM.

Compute utilization measures the percentage of available hardware resources (e.g., CPU cores, NPU threads) used during model execution. High, sustained utilization can cause thermal throttling and impact other device functions.

• Hardware Saturation: Indicates if the model is efficiently leveraging parallel compute units or is bottlenecked by a single thread.
• Co-existence: On-device models must share compute with the device's operating system and other applications; spikey 100% utilization is often undesirable.
• Profiling Tool: Requires low-level profiling tools like perf (Linux) or vendor-specific SDKs for accelerators.

On-device robustness evaluates performance under real-world input variations (e.g., sensor noise, lighting changes). Calibration measures how well the model's predicted confidence scores reflect true likelihoods.

• Expected Calibration Error (ECE): A common metric; a well-calibrated model's "80% confidence" predictions should be correct 80% of the time.

• Edge Cases: Must be tested against locally relevant adversarial conditions (e.g., motion blur for cameras, background noise for microphones).

• Link to Federated Learning: A globally aggregated model may be poorly calibrated for individual clients, necessitating local calibration techniques.

Edge devices have severe limitations in compute, memory, and power. Running full evaluation suites designed for servers is impossible.

Key constraints include:

• Limited RAM: Cannot load large validation datasets or complex evaluation libraries.
• CPU/GPU/ NPU Limitations: Inference is prioritized; evaluation must be lightweight.
• Thermal & Power Budgets: Extended evaluation runs can drain batteries or trigger thermal throttling.

Solutions:

• Streaming Metrics: Calculate metrics like accuracy or F1-score incrementally as data is processed, never storing the full dataset.
• Subset Evaluation: Use a small, statistically representative subset of on-device data for periodic checks.
• Energy-Aware Scheduling: Trigger evaluation during periods of charging or high battery.

A single device holds only a narrow slice of the global data distribution, making local metrics potentially misleading for assessing global model health.

Challenges:

• Local Bias: Metrics reflect only the device's user habits or environment (e.g., a phone in one geographic region).
• Temporal Skew: Data may be non-stationary; today's performance may not predict tomorrow's.

Solutions:

• Cross-Device Validation Protocols: Use secure, privacy-preserving techniques to compute metrics over a distributed validation set held across many devices.
• Synthetic Data Injection: Carefully introduce a small, diverse set of canonical evaluation samples during model deployment to anchor measurements.
• Cohort Analysis: Aggregate and compare metrics from devices with similar profiles (e.g., same model, region) to identify outliers.

Traditional metrics like accuracy are insufficient. On-device evaluation must measure factors critical to the user experience and system constraints.

Essential Edge-Specific Metrics:

• Inference Latency: P95 and P99 latency measured under real device load.
• Power Consumption: Joules per inference, critical for battery-powered devices.
• Model Stability: Memory usage and crash rates during sustained inference.
• Contextual Accuracy: Task success rate in the specific environment (e.g., keyword detection in noisy rooms).

Implementation:

• Lightweight Profilers: Integrate minimal profiling code into the inference pipeline.
• Metric Aggregation: Use techniques like histogram aggregation to compress distributions of latencies for efficient reporting.

To understand global performance, individual device metrics must be aggregated without leaking sensitive information about user activity or local data.

Risks:

• A spike in false positives for a medical diagnostic app could reveal a user's health condition.
• Latency patterns could infer the specific applications a user is running.

Solutions:

• Secure Aggregation: Use cryptographic multi-party computation protocols to sum metrics across devices so the server only sees the aggregate.
• Differential Privacy for Metrics: Add calibrated noise to locally computed metrics before they leave the device, providing formal privacy guarantees.
• Federated Analytics: Apply frameworks like Google's Federated Computation to compute statistics over distributed data without raw data collection.

On-device evaluation is not a one-time event but a continuous process that informs personalization, model selection, and retraining triggers in federated systems.

Workflow Integration:

1. Baseline Establishment: Measure initial model performance post-deployment.
2. Drift Detection: Monitor for concept drift or data drift signaled by decaying local metrics.
3. Personalization Feedback: Use evaluation results (e.g., which classes perform poorly) to guide on-device fine-tuning.
4. Federated Round Contribution: Devices with high-quality, representative data and good local performance can be weighted more heavily in federated averaging.

Tooling: Requires tight integration with the Federated Learning Orchestrator and on-device ML pipelines.

Implementing robust on-device evaluation requires frameworks that abstract hardware complexity and provide standardized primitives.

Leading Frameworks & Capabilities:

• TensorFlow Lite / PyTorch Mobile: Provide micro-benchmarking tools and interpreter options to profile inference.
• MLPerf Tiny: Industry benchmark suite for ultra-low-power devices, providing standardized evaluation workloads.
• OpenFL: An open-source framework for federated learning that includes extensions for federated evaluation protocols.
• Android ML Kit & Core ML: Platform-specific kits often include performance logging and monitoring APIs.

Best Practice: Decouple evaluation code from core model inference to allow toggling it on/off based on device state.

Federated evaluation is the decentralized process of assessing a model's performance across multiple clients without centralizing their private data. This involves:

• Clients computing metrics (e.g., accuracy, loss) locally on their private datasets.
• The secure aggregation of these local metrics (e.g., averages, weighted averages) on a central server to estimate global performance.
• This protocol is essential for validating model improvements in production while maintaining the core privacy guarantees of federated learning.

Local model accuracy is the performance of a model on the private data held by an individual client or device. This metric is distinct from global accuracy because:

• It directly measures how well the model works for that specific user's data distribution.
• It is a key indicator of statistical heterogeneity (non-IID data), as significant variance in local accuracy across clients signals data distribution skew.
• It is the primary metric used in personalized federated learning to tailor models to individual clients.

Cross-client validation is a model evaluation technique that estimates a federated model's ability to generalize to new, unseen data distributions. The process typically involves:

• Training a model on a randomly selected subset of clients.
• Validating the model on the held-out data from a completely different set of clients.
• This method provides a more realistic estimate of performance on novel edge devices than traditional train/test splits on a single dataset, directly addressing the challenge of non-IID data.

Model calibration measures how well a model's predicted probabilities align with true empirical frequencies. A well-calibrated model's confidence is a reliable indicator of correctness. In on-device and federated contexts:

• Calibration is critical for decision-making under uncertainty (e.g., medical diagnostics on edge devices).
• It is measured using metrics like Expected Calibration Error (ECE) or reliability diagrams.
• Federated training on non-IID data can negatively impact calibration, requiring specific techniques to maintain it across diverse clients.

The generalization gap in federated learning quantifies the difference between a model's performance on its local training data and its performance on unseen global data. This gap is critical because:

• It can be significantly larger in federated settings due to client drift and non-IID data.
• A large gap indicates overfitting to local client data, reducing the global model's utility.
• On-device evaluation helps measure this gap by comparing local training metrics to performance on a local validation set or via federated evaluation on other clients' data.

Personalization performance measures the efficacy of techniques that adapt a globally trained federated model to an individual client's specific data distribution. Key aspects include:

• Metrics that compare the accuracy of a personalized model versus the global model on a client's local test set.
• Techniques like fine-tuning, meta-learning, or multi-task learning are used to achieve personalization.
• High personalization performance is often the ultimate goal for on-device models, ensuring the AI works optimally for each unique user and device context.