Partially Observable Markov Decision Process (POMDP)

Since the agent cannot observe the true state, it maintains a probability distribution over all possible states, known as the belief state. This is a sufficient statistic summarizing the entire history of actions and observations.

• The belief state is updated via a Bayesian filter after every action and observation.
• It transforms the POMDP into a continuous-space belief MDP, where the state is the probability vector itself.
• In a factory setting, this represents the agent's confidence that a machine is healthy, degraded, or failed based on vibration sensor readings.

The observation function O(o | s', a) defines the probability of receiving a specific sensor reading given the true hidden state and the action taken. This models sensor noise and ambiguity.

• A thermal camera might report 'overheating' with 95% probability when a bearing is truly failing, but 5% when it is healthy.
• This function captures the partial observability that distinguishes POMDPs from fully observable MDPs.
• In manufacturing, observations are the raw, noisy signals from accelerometers, microphones, and torque sensors.

A POMDP policy maps the agent's belief state to an action, not the true state. This makes policies significantly more complex than MDP policies.

• Policies are often represented as finite-state controllers or alpha-vectors in value function space.
• The optimal policy balances exploitation (taking the best action given current belief) with information gathering (taking actions to reduce uncertainty).
• A maintenance agent might deliberately run a diagnostic cycle—sacrificing throughput—to disambiguate between two failure modes before committing to a repair.

POMDPs formally quantify the value of information—the expected improvement in decision quality from reducing uncertainty. This is a core concept absent from fully observable models.

• An agent can compute whether the cost of a sensor query or diagnostic test is justified by the expected gain in future reward.
• This drives active perception behaviors where the agent chooses actions specifically to improve its observational accuracy.
• In industrial settings, this justifies investments in additional instrumentation by calculating the expected reduction in costly false-positive shutdowns.

Exact POMDP solving is PSPACE-complete for finite horizons and undecidable for infinite horizons in the worst case. Practical solutions rely on approximation.

• Point-based value iteration samples reachable belief points to approximate the value function.
• Monte Carlo methods like POMCP use online planning with particle filters to scale to large state spaces.
• Deep reinforcement learning approaches use recurrent neural networks to implicitly encode belief states from observation histories, bypassing explicit belief tracking.

POMDPs are the natural mathematical framework for condition-based maintenance where equipment degradation is hidden and only indirectly observed through sensor data.

• The true state is the physical health of a component (healthy, worn, cracked, failed).
• Observations are vibration spectra, oil debris counts, and temperature readings.
• The agent decides when to schedule preventive maintenance, balancing the cost of downtime against the risk of catastrophic failure.
• This framework explicitly models the trade-off between running to failure and conservative early replacement.

A probability distribution over all possible true states of the environment, representing the agent's subjective uncertainty about the factory floor condition.

• Core Mechanism: The agent updates this distribution using Bayesian inference after every action and partial observation.
• Continuous Space: The belief state is a continuous probability simplex, making exact POMDP solutions computationally intractable for large state spaces.
• Industrial Example: A robot arm maintains a belief that a component is 'present' with 92% probability and 'missing' with 8% probability based on a noisy laser scan.

The algorithmic component that recursively updates the belief state by fusing action outcomes with noisy sensor observations.

• Bayesian Update Rule: b'(s') = η * O(o|s',a) * Σ T(s'|s,a) * b(s), where η is a normalizing constant.
• Kalman Filter: A closed-form state estimator for linear Gaussian systems, widely used in industrial tracking and navigation.
• Particle Filter: A non-parametric Monte Carlo method that represents the belief state as a set of weighted samples, essential for highly non-linear manufacturing processes.

A machine learning paradigm where an agent learns an optimal policy through trial-and-error interaction, often used to solve POMDPs when transition and observation models are unknown.

• Model-Free RL: Algorithms like Proximal Policy Optimization (PPO) learn directly from experience without building an explicit model of the factory environment.
• Deep Q-Networks (DQN): Use neural networks to approximate the value of actions in high-dimensional belief spaces.
• Partial Observability Challenge: Standard RL fails under partial observability; specialized methods like Deep Recurrent Q-Networks (DRQN) incorporate LSTMs to infer hidden state from observation histories.

The formal engineering discipline of determining whether a system's internal states can be uniquely inferred from its output measurements over time.

• Observability Matrix: A mathematical test used in linear control theory to verify if sensor placement is sufficient to reconstruct the full state vector.
• Sensor Placement Optimization: In a POMDP context, this involves strategically positioning cameras and vibration sensors to minimize belief-state entropy.
• Industrial Relevance: A chemical reactor with only a temperature probe is partially observable; adding a pressure transducer improves observability and reduces the agent's uncertainty.

A statistical model where the system is assumed to be a Markov process with unobservable (hidden) states, closely related to the temporal inference layer of a POMDP.

• Core Difference: HMMs lack an explicit action or control variable; they are purely for inference, whereas POMDPs include an agent that acts to maximize reward.
• Inference Algorithms: The Forward-Backward algorithm and Viterbi algorithm are used to estimate hidden state sequences from observations.
• Manufacturing Use Case: An HMM can model tool wear as a hidden state inferred from surface finish measurements, while a POMDP would also decide when to trigger a tool change.