Proximal Policy Optimization (PPO)

The core innovation of PPO is the clipped surrogate objective function, which prevents destructively large policy updates. Unlike standard policy gradient methods that can take a bad step and permanently degrade performance, PPO clips the probability ratio between the new and old policy to a range of [1-ε, 1+ε] (typically ε=0.2). This creates a trust region that keeps the updated policy close to the previous one, ensuring monotonic improvement. In manufacturing contexts, this prevents an agent controlling a robotic arm from suddenly adopting an erratic movement policy that could damage equipment.

PPO relies on Generalized Advantage Estimation (GAE) to determine how much better a specific action was compared to the expected baseline. The advantage function subtracts the state value from the observed return, reducing variance in the gradient estimate. GAE uses an exponentially weighted average of n-step temporal difference errors, controlled by the λ (lambda) parameter. A λ near 0 reduces variance but introduces bias; a λ near 1 reduces bias but increases variance. For production scheduling, accurate advantage estimation allows the agent to correctly credit a scheduling decision that reduced makespan three steps later.

PPO uses a dual-network actor-critic architecture. The actor network parameterizes the policy π(a|s) and outputs the probability distribution over actions. The critic network estimates the value function V(s), predicting the expected cumulative reward from a given state. During training, the critic's value estimates are used to compute advantages, while the actor is updated using the clipped surrogate objective. The critic is updated to minimize the mean squared error between its predictions and observed returns. This separation allows the agent to simultaneously learn what to do (actor) and how valuable each state is (critic).

Unlike on-policy algorithms that discard data after a single update, PPO performs multiple epochs of stochastic gradient descent on the same collected trajectory data. After gathering a batch of experiences using the current policy, PPO shuffles the data into mini-batches and performs K epochs of updates (typically 3-10). This sample reuse dramatically improves data efficiency without the instability of full off-policy methods. For industrial applications where each interaction with a physical system is costly, this means an agent can learn effective control policies from fewer real-world trials.

PPO adds an entropy bonus term to the loss function to prevent premature convergence to suboptimal deterministic policies. By rewarding the policy for maintaining higher entropy (uncertainty) in its action distribution, the agent continues to explore alternative actions even after finding a reasonable strategy. The entropy coefficient controls the exploration-exploitation trade-off. In manufacturing process control, this prevents an agent from settling on a locally optimal temperature setting and encourages it to explore parameter ranges that might yield higher throughput.

PPO natively supports both continuous and discrete action spaces through different policy head configurations. For continuous control (e.g., setting a CNC machine's spindle speed), the actor outputs the mean and standard deviation of a Gaussian distribution. For discrete control (e.g., selecting which job to schedule next), the actor outputs logits passed through a softmax. This flexibility allows a single algorithm to handle heterogeneous manufacturing tasks—from precise robotic torque control to discrete routing decisions in a flexible manufacturing system.

The mathematical framework that formalizes the environment PPO interacts with. An MDP defines states, actions, a transition probability function, and a reward signal. In manufacturing, a state might be current machine temperature and queue depth; an action might be adjusting feed rate. The Markov property assumes the next state depends only on the current state and action, not the full history. PPO learns a policy that maps states to actions to maximize cumulative discounted reward over a horizon.

The engineering discipline of designing intermediate reward signals to guide PPO agents toward sparse terminal goals. In a robotic assembly task, the terminal reward might be a successful snap-fit, but shaped rewards could include:

• Proximity bonus: distance reduction between gripper and target
• Alignment reward: angular error minimization
• Curiosity bonus: intrinsic reward for visiting novel states Poorly shaped rewards cause reward hacking, where the agent exploits the proxy signal without achieving the true objective. PPO's clipping helps mitigate policy collapse from misspecified rewards.

The dual-network structure at the core of PPO. The actor (policy network) outputs a probability distribution over actions given the current state. The critic (value network) estimates the expected return from that state, serving as a baseline. The advantage is computed as the difference between actual returns and the critic's estimate. PPO updates both networks simultaneously: the actor maximizes the clipped surrogate objective, while the critic minimizes mean squared error against observed returns. This dual learning stabilizes training compared to pure policy gradient methods.

An extension of the MDP framework essential for real-world manufacturing where sensors provide incomplete information. In a POMDP, the agent receives an observation rather than the true state, and must maintain a belief distribution over possible states. PPO can be adapted to POMDPs by feeding a history of observations and actions into a recurrent neural network policy, effectively learning to infer the hidden state. This is critical for tasks like predictive maintenance where internal bearing wear is unobservable but must be inferred from vibration spectra.