Partially Observable MDP (POMDP)

The state space is the set of all possible, hidden configurations of the environment. The agent never directly observes the true state s ∈ S. This is the fundamental source of partial observability. For example, in robot navigation, the true state includes the robot's precise pose and the location of all obstacles, which may be occluded.

• Hidden Variable: The true state s_t is the latent variable the agent must infer.
• Markov Property: The future state depends only on the current state and action, not the full history.

The action space is the set of all actions the agent can execute to influence the environment. Executing action a_t ∈ A at time t causes a transition to a new hidden state s_{t+1} according to the dynamics model and yields a scalar reward r_t. Actions are the agent's only mechanism for interacting with and gathering information about the hidden world.

• Exploration vs. Exploitation: Actions must balance gaining reward with reducing state uncertainty (active perception).

The observation space is the set of possible sensory inputs the agent receives. After taking action a_t, the agent receives an observation o_{t+1} ∈ O that provides noisy, incomplete information about the new state s_{t+1}. The observation model Z(o | s', a) = P(o_{t+1}=o | s_{t+1}=s', a_t=a) defines this probability.

• Sensor Model: This is the agent's 'sensor'. For example, a camera provides a 2D pixel array (observation) of a 3D world (state).
• Corruption: Observations can be noisy, ambiguous, or missing entirely.

The dynamics model (or transition function) T(s' | s, a) = P(s_{t+1}=s' | s_t=s, a_t=a) defines the probability of transitioning to state s' given the current state s and action a. It encodes the rules of the environment. In a POMDP, the agent does not know s, so it cannot predict transitions with certainty. This model is used internally for planning over belief states.

• Stochasticity: Transitions can be probabilistic, adding another layer of uncertainty.

The reward function R(s, a) provides a scalar signal defining the agent's goal. It specifies the immediate reward received for taking action a in state s. The agent's objective is to maximize the expected cumulative discounted reward. In a POMDP, since s is unknown, the agent must compute expected reward based on its belief state b(s).

• Expected Reward: r(b, a) = Σ_s b(s) * R(s, a).

The belief state b_t is a probability distribution over the state space S. It is the agent's internal estimate of the true state, updated recursively using Bayes' rule: b_{t+1}(s') ∝ Σ_s b_t(s) * T(s'|s,a) * Z(o|s',a). The belief state is a sufficient statistic—it summarizes all information from past actions and observations.

• Continuous Space: The belief space B is a continuous space of probability distributions, making POMDPs computationally complex.
• Planning Horizon: The agent plans sequences of actions to optimize future expected reward, starting from its current belief.

A Markov Decision Process (MDP) is the foundational mathematical framework for modeling sequential decision-making in fully observable environments. It is defined by the tuple (S, A, P, R, γ):

• S: A finite set of states.
• A: A finite set of actions.
• P: Transition function, P(s'|s,a), the probability of moving to state s' from state s after taking action a.
• R: Reward function, R(s,a,s'), specifying the immediate reward.
• γ: Discount factor (0 ≤ γ ≤ 1).

The Markov property holds that the future state depends only on the current state and action, not the full history. A POMDP generalizes this framework to environments where the true state is not directly observable.

A belief state is the core mechanism for decision-making in a POMDP. Since the agent cannot observe the true state s, it maintains a probability distribution b(s) over all possible states s ∈ S. This distribution represents the agent's internal estimate of the world.

• Initialization: Starts as a prior distribution.
• Update: After taking action a and receiving observation o, the belief is updated using Bayes' rule: b'(s') ∝ O(o|s',a) Σ_s P(s'|s,a) b(s).
• Sufficient Statistic: The belief state is a sufficient statistic of the history; it contains all information needed to make optimal decisions. The POMDP is thus transformed into a continuous-state MDP over the belief space, where policies map belief states to actions.

Model-Based Reinforcement Learning (MBRL) is a paradigm where an agent learns an explicit model of the environment's dynamics—the transition P and reward R functions—and uses this model for planning. This contrasts with model-free RL, which learns a policy or value function directly from experience.

POMDPs are a formal model-based framework. In practical MBRL (e.g., with pixels), the learned world model must often address partial observability by learning a latent state representation. Algorithms like Dreamer learn a Recurrent State-Space Model (RSSM), which is essentially a learned dynamics model within a latent belief space, enabling planning via imagined rollouts.

State estimation is the algorithmic process of inferring the hidden state of a dynamical system from a sequence of noisy observations and known control inputs (actions). It is the practical engineering counterpart to the belief update in a POMDP.

• Kalman Filter: The optimal estimator for linear Gaussian systems. It maintains a Gaussian belief state (mean and covariance).
• Extended Kalman Filter (EKF): A non-linear extension.
• Particle Filter: A sequential Monte Carlo method that represents the belief state with a set of weighted samples (particles). It is highly effective for non-linear, non-Gaussian problems and is a direct implementation of the POMDP belief update for complex domains like robot localization.

A Recurrent State-Space Model (RSSM) is a specific neural network architecture for learning world models from high-dimensional observations (like pixels) in partially observable environments. It combines:

• A deterministic recurrent path (an RNN) to represent the belief.
• A stochastic latent variable to capture uncertainty about the world.

Used in the Dreamer series of agents, the RSSM learns to encode a history of observations into a compact latent state z_t. This latent state serves as the belief for planning. The model can then predict future latent states and rewards, allowing the agent to conduct planning entirely within its learned latent space, which is computationally efficient and sample-effective.

Model-Predictive Control (MPC) is an online, receding-horizon control method widely used in robotics and process industries. At each time step, it solves a finite-horizon optimization problem using a dynamics model to predict future states, then executes only the first action before replanning.

MPC is a powerful planning algorithm that can be applied within the POMDP framework. When the true state is unknown, MPC operates over the belief space. The optimization considers possible future belief states, often requiring techniques to manage uncertainty, such as using an ensemble of dynamics models to approximate the belief distribution or incorporating explicit uncertainty penalties into the cost function.