Epistemic Uncertainty

Unlike aleatoric uncertainty, epistemic uncertainty can be eliminated by collecting more training data. It is high in regions of the state space where the agent has sparse or no experience. Adding diverse trajectories directly shrinks this uncertainty, making it a signal for active learning and directed exploration.

In Bayesian neural networks and deep ensembles, epistemic uncertainty is quantified by the disagreement between models sampled from the posterior distribution over weights. High variance in Q-value predictions across ensemble members indicates the agent lacks sufficient knowledge to make a confident decision.

Epistemic uncertainty serves as a robust mechanism for novelty detection. When an agent encounters states far from its training distribution, predictive entropy spikes. This allows a policy to trigger a fallback to a safe default action or request human intervention rather than extrapolating blindly.

Total predictive uncertainty is formally decomposed into epistemic + aleatoric components. Aleatoric uncertainty captures inherent environment stochasticity (irreducible noise), while epistemic uncertainty captures model ignorance. Distinguishing them is critical for risk-sensitive domains like autonomous driving.

Agents can be trained to seek states that maximize expected information gain, directly reducing epistemic uncertainty. This drives intrinsically motivated exploration where the agent is rewarded not for external goals but for visiting regions where its world model will learn the most.

In model-based RL, epistemic uncertainty over the learned transition dynamics can be propagated through rollouts. Planning algorithms like PETs (Probabilistic Ensembles with Trajectory sampling) use this to generate cautious action sequences that avoid states where the dynamics model is unreliable.

An internal generative model of the environment learned by an agent. It encodes the agent's beliefs about state transitions, which can be probed and visualized to understand where the agent's knowledge is incomplete. By inspecting the world model's prediction errors, engineers can identify unexplored regions of the state space that contribute directly to high epistemic uncertainty.

A family of off-policy evaluation techniques that estimate how a new policy would perform using historical data. This method explains potential outcomes without deployment by answering 'what if' questions. It is critical for reducing epistemic uncertainty because it allows engineers to estimate performance in data-sparse regions before committing to real-world execution.

A latent state encoding where individual dimensions correspond to independent, meaningful generative factors. When an agent learns a disentangled representation, its internal state becomes inherently interpretable. Engineers can audit which latent factors are unknown or noisy, directly pinpointing the source of epistemic uncertainty in the model's understanding of the environment.

A class of algorithms that model the full probability distribution of returns rather than just the expected value. This approach explicitly captures the agent's uncertainty about outcomes. A wide or bimodal distribution in a novel state signals high epistemic uncertainty, while a sharp distribution indicates confident knowledge, making risk and ignorance directly measurable.

A cognitive model that decomposes the value function into a reward-independent predictive map of future states. This representation separates an agent's knowledge of environmental dynamics from its knowledge of rewards. Epistemic uncertainty can be isolated to the successor map itself, revealing which state transitions the agent has not yet learned reliably.

A causal interpretability method that systematically removes or occludes input features to measure the resulting change in policy output. By ablating features and observing increased prediction variance, engineers can identify which dimensions of the state space the model lacks sufficient data for, providing a direct diagnostic for epistemic uncertainty.