Uncertainty Quantification

Aleatoric uncertainty (or data uncertainty) arises from the inherent randomness, noise, or stochasticity in the data-generating process. It is irreducible, meaning it cannot be reduced by collecting more data. In robotics, this includes:

• Sensor noise from cameras, LiDAR, or inertial measurement units.
• Stochastic dynamics in the environment, like unpredictable object friction or air turbulence.
• Uncontrollable variables such as lighting conditions or the behavior of other agents. Quantifying aleatoric uncertainty is crucial for robust perception and control, as it informs the system about the reliability of its observations. It is often modeled as the variance parameter in a probabilistic neural network's output distribution.

Epistemic uncertainty (or model uncertainty) stems from a lack of knowledge or incomplete information about the system. This type of uncertainty is reducible by gathering more data or improving the model. In simulation-to-real transfer, it manifests as:

• Model misspecification where the simulation's physics engine does not perfectly match real-world dynamics.
• Sparse data in regions of the state-action space not encountered during training.
• Unknown parameters in the system model. Epistemic uncertainty is high for out-of-distribution inputs and decreases as the model learns. Bayesian Neural Networks (BNNs) and ensemble methods like Monte Carlo Dropout are common techniques to quantify it, providing a measure of the model's confidence in its predictions.

Model uncertainty is a specific form of epistemic uncertainty related to the imperfections in the learned policy or predictive model itself. It quantifies how much the model's predictions might change if it were retrained on different data. Key sources include:

• Limited training data leading to underfitting or high variance.
• Architectural choices and hyperparameter settings.
• Approximation errors from function approximators like neural networks. In Safe Reinforcement Learning, model uncertainty is critical for identifying when a robot is operating in an unfamiliar state where its policy is unreliable. Techniques like Bayesian model averaging or deep ensembles explicitly capture this uncertainty to trigger safe fallback behaviors or request human intervention.

Parametric uncertainty refers to uncertainty about the precise values of parameters within a known system model. Even with a correct model structure, parameters may be unknown or vary. In physics-based simulation, this includes:

• Inertial parameters of objects (mass, center of mass, moment of inertia).
• Friction coefficients between contacting surfaces.
• Motor torque constants and actuator delay times. System Identification techniques are used to reduce parametric uncertainty by calibrating simulation models with real-world data. Domain randomization, a core sim-to-real technique, explicitly exposes a policy to a wide distribution of these uncertain parameters during training to build robustness.

Distributional shift occurs when the input data distribution at deployment differs from the distribution the model was trained on, leading to increased uncertainty and performance degradation. It is a primary cause of the reality gap in sim-to-real transfer. Types include:

• Covariate shift: Changes in the input distribution (e.g., different visual textures).
• Label shift: Changes in the distribution of output labels.
• Concept drift: The relationship between inputs and outputs changes over time. Detecting distributional shift is a key safety mechanism. Out-of-Distribution (OOD) detection methods, such as monitoring model uncertainty scores or using dedicated detectors, can identify when a system is operating outside its trained domain, signaling a potential failure mode.

Uncertainty propagation is the process of tracking how initial uncertainties (aleatoric or epistemic) in measurements or model parameters evolve through a system's dynamics or decision-making pipeline. It is fundamental for risk-aware planning. For a robot, this involves:

• Propagating uncertain sensor readings through a perception model.
• Forecasting how state uncertainty grows over a planned trajectory.
• Calculating the probability of constraint violation in a Constrained Markov Decision Process (CMDP). Methods like Kalman filters (for linear-Gaussian systems) and particle filters (for non-linear systems) are classic tools. In deep learning, Bayesian inference or moment matching through neural networks are used to propagate uncertainties for safe, long-horizon decision-making.

UQ distinguishes two fundamental types of uncertainty. Aleatoric uncertainty (or data uncertainty) is inherent randomness in the data-generating process, like sensor noise. It is irreducible. Epistemic uncertainty (or model uncertainty) stems from a lack of knowledge, such as an untrained model encountering an unfamiliar scenario. It is reducible with more data or a better model. In safety, this distinction dictates the response: aleatoric uncertainty may require robust control, while high epistemic uncertainty should trigger a fallback to a safe, conservative mode.

Models can output not just a prediction but a confidence interval or distribution. In anomaly detection, a prediction with high uncertainty (e.g., a wide predictive variance) flags a potential Out-of-Distribution (OOD) sample. For example:

• A medical imaging AI flagging a rare tumor morphology it was not trained on.
• An autonomous vehicle's perception system encountering severe weather conditions absent from its training set. This triggers human intervention or a switch to a verified fallback system, preventing silent failures.

Bayesian Neural Networks are a primary technical approach for UQ. Instead of learning fixed weight parameters, they learn a distribution over possible weights. This allows them to naturally quantify epistemic uncertainty. During inference, multiple forward passes with different sampled weights (Monte Carlo Dropout is a practical approximation) yield a distribution of outputs. The variance of this distribution is a direct measure of model uncertainty. This is critical for assessing confidence in life-or-death decisions, such as a robotic surgical tool's planned trajectory.

Conformal Prediction is a distribution-free, post-hoc framework that provides rigorous, statistically valid uncertainty intervals. Given a desired confidence level (e.g., 95%), it outputs a prediction set guaranteed to contain the true label with that probability, assuming the test data is exchangeable with the calibration data. This provides provable safety guarantees for classification and regression tasks, such as guaranteeing with 99.9% confidence that a predicted aircraft component stress value lies within a calculated range.

In Safe Reinforcement Learning (Safe RL), UQ is used to manage exploration risk. A policy can be trained to be risk-sensitive, penalizing actions leading to high-uncertainty states. Techniques include:

• Using a safety critic that estimates the probability of constraint violation, informed by uncertainty.
• Uncertainty-aware exploration, where the agent actively seeks to reduce epistemic uncertainty in safety-critical parts of the state space.
• Formulating constraints in a Constrained Markov Decision Process (CMDP) using uncertainty-weighted costs.

A Digital Twin is a high-fidelity simulation of a physical asset. For it to be trustworthy for safety validation, the uncertainty in its predictions must be quantified and propagated. This involves:

• Uncertainty propagation through complex physics simulations to understand how input parameter variances affect output metrics (e.g., structural fatigue).
• Sensitivity analysis to identify which uncertain parameters most impact safety margins.
• Calibration against real-world sensor data to reduce epistemic uncertainty in the twin's models, ensuring virtual Hardware-in-the-Loop Testing results are statistically meaningful.

The task of identifying whether a new input data point is statistically different from the training data distribution. This is a critical application of uncertainty quantification for safety.

• Core Function: Flags inputs the model has not seen during training, where predictions are unreliable.
• Methods: Often uses measures of epistemic uncertainty (e.g., high model variance) or density estimation to detect low-probability regions.
• Safety Role: Prevents a model from making confident but erroneous predictions on novel or anomalous inputs, triggering a fallback to a safe policy or human operator.

A subfield of RL focused on developing algorithms that learn to maximize performance while formally satisfying safety constraints during training and deployment.

• Formalization: Typically uses a Constrained Markov Decision Process (CMDP), which adds cost constraints to the standard reward-maximization objective.
• Role of Uncertainty: Algorithms must account for uncertainty in dynamics and outcomes to avoid constraint violations. Uncertainty quantification informs risk-aware exploration.
• Techniques: Include risk-sensitive RL, shielded learning, and the use of safety critics to estimate the probability of entering unsafe states.

The primary mathematical framework for Safe Reinforcement Learning. It extends the standard MDP by incorporating constraints on expected cumulative costs.

• Formal Definition: Defined by the tuple (S, A, P, R, C, d), where C is a cost function and d is a constraint threshold.
• Objective: Find a policy π that maximizes expected cumulative reward, subject to the constraint that expected cumulative cost ≤ d.
• Connection to UQ: Solving a CMDP effectively requires reasoning about aleatoric uncertainty (stochastic outcomes) and epistemic uncertainty (unknown dynamics) to guarantee safety with high probability.

The process of evaluating a model's resilience against deliberately crafted input perturbations designed to cause misclassification or failure.

• Adversarial Examples: Small, often imperceptible perturbations that exploit model blind spots.
• Testing Methodology: Uses optimization techniques (e.g., Projected Gradient Descent) to find the minimal perturbation that causes a failure.
• UQ Link: A model with well-calibrated uncertainty should express high uncertainty for adversarial inputs, providing a diagnostic signal. Robustness is closely tied to a model's sensitivity to distributional shift.

The scenario where the data distribution encountered during a model's deployment differs from the distribution it was trained on, leading to degraded performance.

• Types: Includes covariate shift (input distribution changes), concept shift (input-output relationship changes), and label shift (output distribution changes).
• Primary Safety Risk: The most common cause of failure for deployed ML systems, as models make extrapolations with unknown reliability.
• Mitigation: Uncertainty quantification is the primary tool for detecting shift. Techniques like domain adaptation and continual learning aim to correct for it.

An approach to safe RL where a runtime monitor or verifier (the 'shield') intervenes to override potentially unsafe actions proposed by the learning agent.

• Architecture: The learning agent proposes actions, but a separate safety module filters or replaces them to ensure they keep the system within a formally defined safe set.
• Basis for the Shield: Often uses control barrier functions (CBFs) or pre-computed safe regions. Uncertainty quantification can make the shield more permissive by accounting for stochasticity.
• Advantage: Decouples the learning objective from hard safety guarantees, allowing for efficient learning within safe boundaries.