Bayesian Neural Network (BNN)

Unlike standard neural networks that use point estimates for weights, a Bayesian Neural Network treats each weight as a probability distribution (e.g., a Gaussian). This fundamental shift means the network's output is not a single prediction but a predictive distribution, capturing the model's inherent uncertainty about the correct parameter values given the training data.

BNNs provide a principled decomposition of uncertainty into two key types:

• Aleatoric Uncertainty: Irreducible noise inherent in the observations (e.g., sensor noise).
• Epistemic Uncertainty: Model uncertainty due to limited data, which can be reduced with more training examples. This explicit quantification is vital for robust planning in MBRL, allowing agents to avoid overconfident actions in unfamiliar states.

Training a BNN involves performing Bayesian inference to compute the posterior distribution over weights, p(weights | data), from a prior distribution p(weights) and the likelihood p(data | weights). Since exact inference is intractable for deep networks, approximate methods are used:

• Variational Inference (VI): Approximates the posterior with a simpler, tractable distribution.
• Markov Chain Monte Carlo (MCMC): Uses sampling to approximate the posterior.
• Monte Carlo Dropout: A practical approximation where dropout applied at test time mimics sampling from the posterior.

In Model-Based RL, a BNN is often used as the transition model or reward model. When predicting the next state s_{t+1} = f(s_t, a_t), the BNN outputs a distribution over possible next states. This probabilistic prediction directly informs uncertainty-aware planning algorithms like Pessimistic Exploration, where the agent avoids states with high epistemic uncertainty, or Probabilistic Ensembles, where multiple BNNs model dynamics.

A major challenge in MBRL is compounding error, where small inaccuracies in a deterministic dynamics model explode over long imagined rollouts. BNNs address this by providing uncertainty estimates that grow with prediction horizon. Planning algorithms can use this signal to truncate rollouts or down-weight trajectories that venture into highly uncertain regions of the state space, leading to more robust long-horizon behavior.

The primary trade-off for BNNs is computational cost. Making predictions requires marginalization over the weight posterior, typically approximated by drawing multiple samples (forward passes). This makes inference slower than a standard forward pass. However, this cost is often justified in MBRL for the gains in sample efficiency and safety, as the agent can learn an effective policy with fewer, more informative interactions with the real environment.

The process of estimating both epistemic uncertainty (from lack of data) and aleatoric uncertainty (inherent randomness) in a model's predictions. In MBRL, this is critical for robust planning. BNNs provide a principled, Bayesian framework for this quantification by representing weights as probability distributions, directly informing exploration strategies and risk-aware decision-making.

A technique for model-based RL where multiple neural networks (e.g., 5-10) are trained independently on the same data to form a dynamics model ensemble. The disagreement (variance) among ensemble members serves as a proxy for predictive uncertainty. This is a frequentist alternative to BNNs for estimating epistemic uncertainty, often used in algorithms like PETS (Probabilistic Ensembles with Trajectory Sampling) for planning.

Also known as conservative model-based RL, this is a strategy where an agent's policy is constrained to avoid states where the learned model is highly uncertain. This is crucial for offline RL and safe deployment. BNNs naturally support this by providing a distribution over model predictions; planners can then sample pessimistic trajectories or penalize actions leading to high-variance future states, preventing the exploitation of model errors.

• Model Error: The discrepancy between a learned dynamics model's predictions and the true environment. It is the primary challenge in MBRL.
• Compounding Error: The catastrophic accumulation of small model inaccuracies over long imagined rollouts, leading to unrealistic simulated states and poor policy learning. BNNs help mitigate this by allowing planners to reason about uncertainty over long horizons, potentially truncating rollouts where uncertainty exceeds a threshold.

A model that learns to predict future states in a compressed, abstract latent space rather than the high-dimensional raw observation space (e.g., pixels). This improves generalization and computational efficiency. Architectures like the Recurrent State-Space Model (RSSM) used in the Dreamer algorithm often incorporate stochastic latent variables, sharing a Bayesian probabilistic motivation with BNNs for representing uncertainty in the latent state transitions.

An agent's internal, learned representation that simulates the environment's dynamics and reward structure. It enables planning and imagination without real interaction. A BNN can serve as the core of a probabilistic world model, where its predictive distributions allow the agent to simulate not just a single future, but a distribution of plausible futures, leading to more robust planning and a better understanding of risk.