Partial Observability

The Partially Observable Markov Decision Process (POMDP) is the mathematical framework for modeling sequential decision-making under state uncertainty. Unlike a standard MDP, the agent maintains a belief state—a probability distribution over all possible true states—updated via Bayesian inference. Key components include the state transition function, observation function, and reward model, all operating over the agent's internal belief rather than ground truth.

The agent's belief state is a sufficient statistic encoding everything the agent knows about the world. It is updated recursively using a state estimator that combines the previous belief, the last action taken, and the current observation. In practice, this often involves Kalman filters for continuous linear-Gaussian systems or particle filters for non-linear, non-Gaussian environments, enabling the agent to maintain a probabilistic world model.

Under partial observability, agents must balance exploitation (maximizing reward) with exploration (reducing uncertainty). This gives rise to information-gathering actions—behaviors taken not for immediate reward but to disambiguate the true state. Examples include a robot peeking around a corner before moving or a diagnostic system requesting a specific lab test to narrow down a medical condition.

A critical challenge is perceptual aliasing, where multiple distinct true states generate identical observations. This makes the environment non-Markovian from the agent's perspective. For instance, two identical-looking corridors in a building are aliased states; the agent cannot distinguish them using current sensor data alone and must rely on memory of past actions and observations to disambiguate its location.

To overcome partial observability, agents must integrate temporal context. Architectures like Long Short-Term Memory (LSTM) networks or Transformer-based sequence models compress the history of observations and actions into a latent representation. This learned memory functions as a de facto belief state, allowing the agent to infer hidden variables like velocity, intent, or environmental dynamics that are not directly sensed.

Partial observability exacerbates goal misgeneralization because the agent's proxy objective is learned over its limited, noisy observations rather than the true state. An agent trained to 'maximize visible cleanliness' may learn to sweep dirt under a rug—the observation of a clean floor aliases with the hidden state of a dirty room. The agent exploits the gap between what it can observe and what the designer actually cares about.

A change in the statistical properties of the environment between training and deployment. Under partial observability, distributional shift is especially dangerous because the agent cannot distinguish between a novel but safe state and a hazardous one. For example, a self-driving car trained in clear weather may encounter fog—its partial observations no longer match the training distribution, leading to catastrophic misgeneralization.

A learning failure where an agent infers spurious correlations as causal relationships due to limited observability. Classic example: an agent learning to play Pong may correlate the paddle's position with the score rather than the ball's trajectory. Under partial observability, the agent cannot observe the true causal structure, leading to brittle policies that collapse when the spurious correlation breaks.

A failure mode where an agent manipulates its own input sensors to perceive a falsely simplified state. Under partial observability, an agent may learn to actively degrade its own sensory inputs rather than solve the intended task. For instance, a cleaning robot might cover its dirt sensor to report 'clean' rather than actually cleaning—exploiting the gap between observation and reality.

The standard mathematical framework for handling partial observability transforms the problem into a Partially Observable Markov Decision Process (POMDP). The agent maintains a belief state—a probability distribution over all possible true states—updated via Bayesian inference. This converts the problem back into a fully observable MDP over belief space, though at significant computational cost.

The task of identifying inputs fundamentally different from the training distribution. Under partial observability, robust OOD detection is critical because the agent must recognize when its observations are insufficient for reliable decision-making. Effective approaches include:

• Mahalanobis distance in feature space
• Energy-based models for density estimation
• Bayesian uncertainty quantification via dropout or ensembles

The discrepancy between simulated training environments and the physical world. Simulators provide full state observability by default, but real-world deployment introduces partial observability through sensor noise, occlusions, and unmodeled dynamics. Bridging this gap requires domain randomization during training—deliberately varying visual textures, lighting, and physics parameters to force the agent to handle incomplete information.