Markov Decision Process (MDP)

The exhaustive set of all possible configurations the manufacturing environment can assume at any time step. In a fully observable MDP, the agent has complete access to the true state.

• Fully Observable: The agent sees the exact factory floor status, including machine temperatures, queue lengths, and tool wear.
• Discrete vs. Continuous: States can be categorical (e.g., machine idle, running, faulted) or continuous vectors (e.g., pressure readings, spindle speeds).
• Markov Property: The future depends only on the current state and action, not the history of how the agent arrived there.

The complete set of decisions available to the agent at each time step. Actions transition the environment from one state to the next, potentially stochastically.

• Primitive Actions: Low-level commands like setting a motor voltage, opening a valve, or triggering a gripper.
• Macro-Actions: Higher-level options such as dispatching a job to a specific CNC machine or rerouting an AGV to a charging station.
• Constraint Adherence: The action space must respect physical limits, safety interlocks, and process tolerances defined by the manufacturing execution system.

Defines the probability of moving from state s to state s' after taking action a. This captures the inherent stochasticity of physical manufacturing processes.

• Stochastic Outcomes: Drilling a hole might result in a perfect bore (99.2% probability) or a minor burr (0.8% probability) due to tool vibration.
• Deterministic Subsystems: Some components, like digital logic controllers, may have near-deterministic transitions for modeling simplicity.
• Model-Based Learning: If T is known a priori from physics simulations or historical data, the agent can plan offline using dynamic programming.

A scalar signal the environment returns after each action, defining the immediate desirability of a state-action pair. The agent's sole objective is to maximize the cumulative sum of these signals over time.

• Sparse Rewards: A +1000 bonus for completing a production batch, with zero reward elsewhere. Requires sophisticated credit assignment.
• Dense Shaping: Incremental rewards for reducing cycle time (-1 per second), minimizing scrap (-50 per defect), or conserving energy (-0.01 per watt).
• Penalties: Negative rewards for safety violations, machine crashes, or deadline misses encode hard constraints into the objective function.

A coefficient between 0 and 1 that determines the present value of future rewards. It mathematically bounds the infinite sum and encodes the agent's planning horizon.

• γ ≈ 0: A myopic agent that greedily maximizes immediate throughput, ignoring long-term tool wear or maintenance scheduling.
• γ ≈ 1: A far-sighted agent willing to sacrifice short-term speed to optimize total productive maintenance cycles and overall equipment effectiveness.
• Economic Interpretation: In supply chain MDPs, γ often maps directly to the risk-free interest rate, discounting future cash flows from fulfilled orders.

The agent's strategy—a mapping from states to actions (deterministic) or to probability distributions over actions (stochastic). The goal of solving an MDP is to find the optimal policy π*.

• Deterministic Policy: In state s, always execute action a. Simple to deploy on a PLC but may get stuck in deterministic loops.
• Stochastic Policy: In state s, choose action a1 with 70% probability and a2 with 30%. Useful for exploration and game-theoretic scenarios.
• Value-Based Methods: Derive π* indirectly by first learning the optimal Q-values for each state-action pair, then acting greedily.

An MDP agent continuously selects cutting parameters—spindle speed, feed rate, and depth of cut—based on real-time sensor feedback. The state space captures tool wear, vibration signatures, and thermal conditions. The reward function balances material removal rate against tool life preservation, learning policies that reduce cycle time by 12-18% while extending tool longevity.

MDPs model the factory floor as a state space of machine availability, job queues, and order priorities. At each decision epoch, the agent selects which job to dispatch to which machine. The reward function penalizes tardiness and setup time while rewarding on-time delivery. Unlike static heuristics, the MDP policy adapts to stochastic events like machine breakdowns and rush orders.

The MDP state encodes equipment health indicators: vibration spectra, oil debris counts, and thermal gradients. The agent chooses between continue operation, schedule inspection, or immediate shutdown. The reward balances production uptime against catastrophic failure cost. The optimal policy triggers intervention only when the expected cost of failure exceeds the cost of preventive action.

MDPs optimize the timing and composition of batch processes—such as heat treatment or chemical curing—against dynamic electricity pricing. The state includes current energy tariff, batch queue length, and deadline constraints. The agent learns to shift energy-intensive operations to off-peak windows while respecting work-in-process holding costs and customer due dates.

Each AGV operates as an MDP agent navigating a grid-world representation of the factory floor. The state captures current position, battery level, and congestion hotspots. Actions include movement in cardinal directions and charging station visits. The reward function minimizes delivery latency while penalizing battery depletion and collision risk, enabling decentralized fleet coordination.

An MDP agent determines the optimal sampling frequency and inspection depth for each production batch. The state tracks historical defect rates, supplier quality scores, and inspection resource availability. The reward balances inspection cost against the expected cost of escapes—defective units reaching customers. The policy dynamically tightens sampling when upstream signals indicate elevated risk.