Bayesian Neural Network (BNN)

Training a BNN involves performing Bayesian inference to compute the posterior distribution over weights, p(weights | data), from a prior distribution p(weights) and the likelihood p(data | weights). Since exact inference is intractable for deep networks, approximate methods are used:

• Variational Inference (VI): Approximates the posterior with a simpler, tractable distribution.
• Markov Chain Monte Carlo (MCMC): Uses sampling to approximate the posterior.
• Monte Carlo Dropout: A practical approximation where dropout applied at test time mimics sampling from the posterior.

In Model-Based RL, a BNN is often used as the transition model or reward model. When predicting the next state s_{t+1} = f(s_t, a_t), the BNN outputs a distribution over possible next states. This probabilistic prediction directly informs uncertainty-aware planning algorithms like Pessimistic Exploration, where the agent avoids states with high epistemic uncertainty, or Probabilistic Ensembles, where multiple BNNs model dynamics.

A major challenge in MBRL is compounding error, where small inaccuracies in a deterministic dynamics model explode over long imagined rollouts. BNNs address this by providing uncertainty estimates that grow with prediction horizon. Planning algorithms can use this signal to truncate rollouts or down-weight trajectories that venture into highly uncertain regions of the state space, leading to more robust long-horizon behavior.

The primary trade-off for BNNs is computational cost. Making predictions requires marginalization over the weight posterior, typically approximated by drawing multiple samples (forward passes). This makes inference slower than a standard forward pass. However, this cost is often justified in MBRL for the gains in sample efficiency and safety, as the agent can learn an effective policy with fewer, more informative interactions with the real environment.

Also known as conservative model-based RL, this is a strategy where an agent's policy is constrained to avoid states where the learned model is highly uncertain. This is crucial for offline RL and safe deployment. BNNs naturally support this by providing a distribution over model predictions; planners can then sample pessimistic trajectories or penalize actions leading to high-variance future states, preventing the exploitation of model errors.

• Model Error: The discrepancy between a learned dynamics model's predictions and the true environment. It is the primary challenge in MBRL.
• Compounding Error: The catastrophic accumulation of small model inaccuracies over long imagined rollouts, leading to unrealistic simulated states and poor policy learning. BNNs help mitigate this by allowing planners to reason about uncertainty over long horizons, potentially truncating rollouts where uncertainty exceeds a threshold.

A model that learns to predict future states in a compressed, abstract latent space rather than the high-dimensional raw observation space (e.g., pixels). This improves generalization and computational efficiency. Architectures like the Recurrent State-Space Model (RSSM) used in the Dreamer algorithm often incorporate stochastic latent variables, sharing a Bayesian probabilistic motivation with BNNs for representing uncertainty in the latent state transitions.

An agent's internal, learned representation that simulates the environment's dynamics and reward structure. It enables planning and imagination without real interaction. A BNN can serve as the core of a probabilistic world model, where its predictive distributions allow the agent to simulate not just a single future, but a distribution of plausible futures, leading to more robust planning and a better understanding of risk.