Uncertainty Quantification

Uncertainty in machine learning is categorized into two fundamental types. Aleatoric uncertainty is inherent randomness or noise in the data itself (e.g., sensor noise, unpredictable environmental effects). It is irreducible. Epistemic uncertainty stems from a lack of knowledge or model ignorance, often due to insufficient training data or operating outside the training distribution. It is reducible with more data or a better model. In sim-to-real transfer, distinguishing between them is crucial: high epistemic uncertainty signals the policy is in an unfamiliar state, requiring caution.

A Bayesian Neural Network is a probabilistic model that represents uncertainty by placing probability distributions over its weights, rather than using fixed point estimates. This allows the network to express predictive uncertainty directly. During inference, predictions are made by sampling from the weight distributions, yielding a distribution of possible outputs. Key techniques include:

• Variational Inference: Approximates the complex true posterior over weights with a simpler distribution.
• Monte Carlo Dropout: A practical approximation where dropout is applied at test time to generate multiple stochastic predictions. BNNs are a core method for quantifying epistemic uncertainty in deep learning policies.

Ensemble methods quantify uncertainty by training multiple models (an ensemble) with different initializations or on different data subsets. The variance in the predictions across the ensemble members serves as a measure of model uncertainty. For a robotic policy, this involves:

• Training N independent policies on the same simulation task.
• At deployment, querying all policies for a given state.
• Using the mean of the action predictions as the final command.
• Using the variance (or standard deviation) as the uncertainty signal. High variance indicates states where the models disagree, highlighting epistemic uncertainty and potential sim2real gap issues.

Conformal prediction is a distribution-free, post-hoc framework for generating statistically rigorous prediction sets with guaranteed coverage. It provides calibrated uncertainty intervals. The process is:

1. Train a base model (any ML model).
2. Define a nonconformity score measuring how unusual a prediction is.
3. Calculate scores on a held-out calibration set.
4. For a new input, create a prediction set containing all labels with a nonconformity score below a chosen threshold, guaranteeing (e.g., 95%) that the true label is within the set. In robotics, this can produce action sets or state prediction regions with formal error bounds, enabling fail-safes.

Uncertainty quantification acts as a safety monitor during policy transfer. The core workflow is:

• Train a policy in simulation with a UQ-capable model (e.g., BNN, Ensemble).
• Deploy on real hardware with the UQ module active.
• Monitor the predictive uncertainty (especially epistemic) in real-time.
• Trigger Safeguards when uncertainty exceeds a predefined threshold. Safeguards can include:
  • Falling back to a conservative, hand-crafted controller.
  • Entering a safe stop state.
  • Requesting human intervention.
  • Initiating targeted online data collection for adaptation. This creates a robust deployment pipeline where the system knows what it doesn't know.

Evaluating the quality of uncertainty estimates is as important as generating them. Key metrics include:

• Calibration: A model is well-calibrated if its predicted confidence scores match the actual frequency of correctness. For example, when a model predicts with 90% confidence, it should be correct 90% of the time. Expected Calibration Error (ECE) is a common metric.
• Sharpness: Refers to the concentration of the predictive distribution. A good UQ method is both calibrated (reliable) and sharp (precise).
• Proper Scoring Rules: Metrics like Negative Log-Likelihood (NLL) or Brier Score evaluate probabilistic forecasts by penalizing both overconfidence and underconfidence. Poorly calibrated uncertainty can lead to false confidence, making calibration a critical step before deployment.

The primary application in sim-to-real transfer is identifying when a policy is operating outside its trained domain. Epistemic uncertainty (model uncertainty) spikes when the robot encounters novel observations not represented in the simulation training data. This acts as a real-time out-of-distribution (OOD) detector, flagging scenarios where the policy's actions are unreliable. For example, a robot trained in simulation might see low uncertainty on clean lab floors but high uncertainty on a newly encountered patterned carpet, signaling a potential dynamics mismatch or observation space mismatch.

During online adaptation or fine-tuning on real hardware, uncertainty estimates guide data collection. A robot can be programmed to explore states with high uncertainty to efficiently gather informative data that reduces model ignorance. This active learning approach minimizes the amount of risky, real-world trial-and-error needed. Techniques like Bayesian optimization use uncertainty to balance exploration (trying uncertain actions) with exploitation (using known good actions), accelerating safe policy adaptation.

Policies can be modified to incorporate uncertainty directly into their action selection, leading to more conservative and robust behavior. Methods include:

• Uncertainty-weighted rewards: Penalizing actions taken in high-uncertainty states.
• Chance-constrained optimization: Ensuring the probability of violating a safety constraint remains below a threshold, using uncertainty estimates.
• Robust Model Predictive Control (MPC): Solving control problems over a distribution of possible dynamics models, accounting for aleatoric uncertainty (inherent noise) in predictions.

Uncertainty quantification is integral to system identification. When calibrating a simulation's physical parameters (e.g., friction coefficients, motor gains) to match real-world data, Bayesian methods provide not just point estimates but full posterior distributions. This reveals which parameters are well-identified by the data and which remain highly uncertain, guiding engineers on where to invest effort in gathering more informative data to reduce simulation bias and narrow the reality gap.

Uncertainty metrics provide a quantitative basis for sim-to-real benchmarking. A policy that transfers with consistently low predictive uncertainty across a range of real-world tests is more robust than one with high, variable uncertainty. This is critical for safety and failure mode simulation, where policies are stress-tested in virtual edge cases. Monitoring uncertainty during shadow mode deployment provides a leading indicator of potential failures before a policy takes full control.

Uncertainty quantification naturally arises from using policy ensembles. The disagreement (variance) among an ensemble's predictions is a powerful measure of epistemic uncertainty. This can be used to select the most confident policy from a set or to trigger a fallback to a safer, rule-based controller. Furthermore, during policy distillation, the uncertainty of the teacher ensemble can be distilled into the student, providing a more informative training signal than just the mean action.

The reality gap, or sim2real gap, is the performance discrepancy between a policy's behavior in simulation and its behavior on physical hardware. This gap is caused by systematic inaccuracies in the simulated model, including:

• Dynamics Mismatch: Differences in simulated vs. real physics (friction, inertia, contact forces).
• Observation Space Mismatch: Differences between perfect simulation state vectors and noisy, delayed real-world sensor data.
• Simulation Bias: Inherent simplifications in rendering, sensor models, or actuator dynamics. Uncertainty quantification directly targets this gap by identifying when a policy is operating in conditions not well-represented during simulation training.

Domain adaptation is a machine learning technique that improves a model's performance on a target domain (the real world) by leveraging knowledge from a related source domain (simulation), despite differences in their data distributions. Key methods include:

• Adversarial Adaptation/Domain-Adversarial Training: Uses a minimax game to learn domain-invariant features that confuse a classifier trying to distinguish source from target data.
• Addressing Covariate Shift: Where the input feature distribution changes between simulation and reality, but the conditional output distribution remains stable. This is a core strategy for closing the reality gap, often used in conjunction with uncertainty estimates to guide the adaptation process.

Online adaptation refers to the real-time adjustment of a policy's parameters or behavior based on streaming data from the environment during execution. This allows a robot to cope with unforeseen changes, wear and tear, or environmental dynamics not captured in simulation.

• Contrast with Offline Adaptation: Online adaptation uses live sensorimotor data, while offline adaptation uses a static pre-collected dataset.
• Role of Uncertainty: High predictive uncertainty can trigger more aggressive online learning or a fallback to safer, more conservative control modes.
• Example: A legged robot adjusting its gait in real-time upon detecting slippery terrain, guided by elevated uncertainty in its contact model.

Policy robustness is the property of a control policy to maintain acceptable performance despite variations in the environment, such as sensor noise, actuator delays, or physical perturbations. It is the primary goal of sim-to-real transfer techniques.

• Achieved through: Domain randomization, robust reinforcement learning algorithms, and policy ensembles.
• Quantified by: Performance degradation under a set of randomized test conditions or adversarial disturbances.
• Link to Uncertainty: A robust policy should exhibit low epistemic uncertainty within its trained domain of operation. High uncertainty often correlates with a drop in robustness, signaling the policy is in unfamiliar territory.

Shadow mode deployment is a safe rollout strategy where a new policy runs in parallel with the existing production system. It processes real-world sensor data and generates predicted actions 'in the shadows' but does not actuate the physical system.

• Primary Purpose: To validate policy performance and gather crucial real-world data for offline analysis without any risk.
• Critical for UQ: The logged differences between the shadow policy's actions and the legacy system's actions, combined with the policy's own uncertainty estimates, provide a rich dataset for identifying failure modes and domain gaps.
• Precursor to Live Deployment: Successful shadowing, with low uncertainty and high agreement with safe actions, builds confidence for a full hardware takeover.

System identification is the process of building or refining a mathematical model of a physical system (e.g., a robot's dynamics) by analyzing its input-output data. In sim-to-real, it's used to calibrate a simulation to better match real-world behavior.

• Reduces Dynamics Mismatch: By tuning simulation parameters (mass, friction coefficients) to minimize the error between simulated and real robot trajectories.
• Data-Driven: Relies on data collected from the real system, often through specific excitation trajectories.
• Synergy with UQ: A well-identified system model reduces epistemic uncertainty. The residual uncertainty after identification often points to unmodeled nonlinearities or stochastic effects, guiding further model refinement.