Reinforcement Learning Agent

The policy is the agent's brain—a mapping from perceived environmental states to actions. It defines the agent's behavior at a given time.

• Deterministic Policy: Maps a state directly to a specific action a = μ(s). Common in Deep Deterministic Policy Gradient (DDPG) algorithms for continuous control.
• Stochastic Policy: Outputs a probability distribution over actions π(a|s). Essential for exploration in Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC).
• Parameterization: In deep RL, the policy is a neural network with weights θ, denoted as π_θ. The goal of learning is to find the optimal weights θ* that maximize expected cumulative reward.

The value function predicts the expected long-term return, acting as a critic that evaluates how good a state or action is.

• State-Value Function V(s): The expected cumulative reward starting from state s and following policy π thereafter. Answers: "How good is it to be in this state?"
• Action-Value Function Q(s,a): The expected return starting from state s, taking action a, and then following policy π. Answers: "How good is it to take this action from this state?"
• Bellman Equation: The recursive mathematical backbone that expresses the relationship between the value of a current state and the values of successor states, enabling iterative learning via temporal-difference updates.

All goals can be framed as the maximization of the expected value of the cumulative reward signal. The reward R_t is a scalar feedback signal sent by the environment at each time step.

• Dense vs. Sparse Rewards: Dense rewards provide frequent feedback (e.g., distance to target), while sparse rewards only signal task completion. Sparse rewards are often more natural but significantly harder to learn from.
• Reward Shaping: The engineering practice of designing intermediate reward functions to guide the agent toward desired behavior without introducing unintended loopholes or reward hacking.
• Discounted Return: Future rewards are geometrically decayed by a discount factor γ ∈ [0,1]. A γ close to 0 makes the agent myopic; a γ close to 1 makes it farsighted, valuing long-term outcomes.

The fundamental dilemma of decision-making under uncertainty. The agent must exploit known high-reward actions while exploring unknown actions that might yield even higher returns.

• ε-Greedy: A simple strategy where the agent picks a random action with probability ε, and the greedy action otherwise. ε typically decays over time.
• Upper Confidence Bound (UCB): Selects actions based on their potential upside, balancing the estimated Q-value with an exploration bonus proportional to the uncertainty of that estimate.
• Entropy Regularization: Used in modern algorithms like SAC, this adds a bonus reward proportional to the entropy of the policy, explicitly encouraging the agent to maintain stochasticity and explore diverse strategies.

The formal mathematical framework that defines the agent-environment interface. An MDP is a tuple (S, A, P, R, γ).

• Markov Property: The future is conditionally independent of the past given the present. A state S_t is Markov if it captures all relevant information from history.
• Transition Dynamics P(s'|s,a): The probability of transitioning to state s' after taking action a in state s. In model-free RL, the agent does not have access to this function.
• Partially Observable MDP (POMDP): An extension where the agent receives an observation O_t that is a noisy or incomplete proxy for the true underlying state, requiring memory mechanisms like LSTMs or transformers.

A critical architectural distinction based on whether the agent learns or uses a model of the environment's dynamics.

• Model-Free: Learns a policy or value function directly from raw experience tuples (s, a, r, s') without predicting future states. Examples: Q-Learning, DDPG. High asymptotic performance but often sample-inefficient.
• Model-Based: Learns an explicit transition model P(s'|s,a) and reward function R(s,a). The agent can then "imagine" future trajectories via planning or model-predictive control (MPC). Dramatically improves sample efficiency but suffers from model bias.
• Dyna Architecture: A hybrid approach that uses real experience to learn a model and then generates simulated experience from that model to accelerate model-free policy learning.

A model-free, off-policy actor-critic algorithm that concurrently learns a Q-function and a deterministic policy. DDPG is specifically designed for continuous action spaces—such as adjusting valve positions or motor torques—where discretizing actions would be impractical. It employs experience replay to break temporal correlations and soft target updates to stabilize deep neural network training. In manufacturing, DDPG enables an agent to learn smooth, precise control policies for tasks like robotic assembly or chemical dosing without requiring a pre-existing process model.

An advanced control algorithm that uses an explicit dynamic process model to predict future plant outputs over a finite receding horizon. At each time step, MPC solves a constrained optimization problem to compute the optimal sequence of control moves, but only implements the first move. This contrasts with RL agents, which learn a policy through interaction rather than relying on a pre-specified model. Hybrid approaches often use RL to tune the cost function weights of an MPC or to learn the system dynamics model itself.

A sim-to-real transfer technique critical for training RL agents in simulation before deploying them on physical manufacturing hardware. During training, the simulator's visual and physical parameters—such as lighting, friction coefficients, camera noise, and part dimensions—are randomized at the start of each episode. This forces the agent to learn a policy that is robust to the reality gap, generalizing to the variability of the real factory floor without requiring an impossibly accurate digital twin. It is essential for vision-based robotic grasping and assembly tasks.

A sequential design strategy for optimizing expensive black-box objective functions. It builds a probabilistic surrogate model—often a Gaussian Process—of the objective and uses an acquisition function to intelligently select the next set of parameters to evaluate. In the context of RL agents, Bayesian optimization is frequently used for automatic hyperparameter tuning of the agent's learning rate, discount factor, and network architecture, as well as for directly optimizing process parameters where running physical experiments is costly and time-consuming.

An optimal recursive algorithm that estimates the internal state of a linear dynamic system from a series of noisy sensor measurements. It operates in a two-step cycle: predict the state and uncertainty forward using a process model, then update the estimate by weighting the prediction against a new measurement based on their relative uncertainties. RL agents often rely on Kalman filters for state estimation, providing a cleaned, low-latency signal of the true process state—such as true temperature or vibration—from corrupted sensor data before the agent decides on a control action.

The field of building mathematical models of dynamic systems from observed input-output data. Techniques range from classical linear methods like ARX and state-space subspace identification to non-linear approaches using neural networks. For model-based RL agents, system identification provides the initial environment dynamics model that the agent uses for planning or as a prior. In adaptive control, online system identification runs continuously to update the model as tool wear or feedstock changes alter the plant's behavior, ensuring the agent's policy remains optimal.