Reinforcement Learning (RL)

The agent is the autonomous decision-maker that learns a policy to maximize cumulative reward. It is typically implemented as a software algorithm, such as a neural network, that maps observations from the environment to actions.

• Core Function: Executes a policy (π) to select actions.
• Learning Mechanism: Updates its policy based on experience using algorithms like Q-Learning or Policy Gradient Methods.
• Example: In a game-playing system, the agent is the AI player that decides which move to make next.

The environment is the external world with which the agent interacts. It receives the agent's actions, transitions to a new state, and emits a scalar reward signal and an observation.

• State (s): The complete description of the environment at a given time.
• Observation (o): The partial or noisy information about the state that the agent actually perceives.
• Dynamics: Governed by a transition function P(s'|s, a) that defines the probability of moving to state s' after taking action a in state s.
• Example: For a warehouse robot, the environment is the physical warehouse layout, including shelves, packages, and other robots.

These spaces define the set of all possible situations the agent can be in and all moves it can make.

• State Space (S): The set of all possible states of the environment. Can be discrete (e.g., board game positions) or continuous (e.g., joint angles of a robot arm).
• Action Space (A): The set of all possible actions the agent can take. Also discrete (e.g., move left/right) or continuous (e.g., apply torque values).
• Dimensionality: High-dimensional spaces (like raw pixels) require function approximation with deep neural networks, leading to algorithms like Deep Q-Networks (DQN).

The reward function R(s, a, s') provides the agent with a scalar feedback signal that defines the goal of the task. It is the primary mechanism for shaping desired behavior.

• Design Challenge: A poorly shaped reward can lead to unintended or reward-hacking behaviors. Reward shaping is often used to provide denser, more informative signals.
• Objective: The agent's goal is to maximize the expected cumulative reward, often a discounted sum: G_t = Σ γ^k * R_{t+k+1}, where γ (gamma) is the discount factor (0 ≤ γ ≤ 1).
• Example: A +1 reward for reaching a goal, a -0.01 penalty for each time step to encourage speed, and a -10 penalty for crashing.

The policy is the agent's strategy, defining the probability distribution over actions given a state. It is the function the agent learns and refines.

• Stochastic Policy: π(a|s) = probability of taking action a in state s. Common in Policy Gradient Methods.
• Deterministic Policy: a = μ(s), a direct mapping from state to action.
• Optimization: The policy is directly optimized (policy-based methods) or derived from a learned value function (value-based methods). Actor-Critic Architectures combine both approaches.

Value functions estimate the long-term desirability of states or state-action pairs, guiding the agent toward higher future reward.

• State-Value Function Vπ(s): The expected return when starting in state s and following policy π thereafter.
• Action-Value Function Qπ(s, a): The expected return after taking action a in state s and thereafter following policy π. The core of Q-Learning.
• Bellman Equation: These functions satisfy recursive Bellman equations, which form the basis for Temporal Difference (TD) Learning algorithms that update estimates based on other estimates.

This primary distinction defines whether an agent learns an explicit model of the environment's dynamics.

• Model-Free RL: The agent learns a policy or value function directly from interaction with the environment, without constructing an internal model of state transitions or rewards. Examples include Q-Learning, Policy Gradient methods, and Actor-Critic architectures. It is often more flexible but can be sample-inefficient.
• Model-Based RL: The agent first learns (or is given) a model that predicts the next state and reward given the current state and action. This model is then used for planning (e.g., via Model Predictive Control) or to generate simulated data to improve policy learning. This approach aims for greater sample efficiency but can suffer from model bias if the learned dynamics are inaccurate.

These algorithms learn the value of states or state-action pairs, deriving an optimal policy implicitly by selecting actions that maximize the estimated value.

• Core Mechanism: They iteratively approximate the optimal action-value function Q(s,a)* or state-value function V(s)* using the Bellman equation.
• Policy: The policy is typically greedy or epsilon-greedy with respect to the learned Q-values.
• Examples: Q-Learning (tabular), Deep Q-Network (DQN) (with neural network function approximation), and its variants like Double DQN and Dueling DQN.
• Characteristics: Often off-policy, discrete action spaces, can be less stable with function approximation due to moving targets.

These algorithms directly parameterize and optimize the policy function π(a|s; θ) that maps states to actions (or action distributions).

• Core Mechanism: They use gradient ascent on the expected cumulative reward with respect to the policy parameters θ. The gradient is estimated via methods like the REINFORCE algorithm.
• Advantages: Naturally handle continuous action spaces, can learn stochastic policies, and often exhibit better convergence properties in high-dimensional spaces.
• Examples: Vanilla Policy Gradient (REINFORCE), Trust Region Policy Optimization (TRPO), and Proximal Policy Optimization (PPO).
• Characteristics: Typically on-policy, can have high variance in gradient estimates.

A hybrid architecture that combines the strengths of value-based and policy-based approaches, forming the foundation for most modern, state-of-the-art algorithms.

• Components:
  • Actor: A policy network that selects actions.
  • Critic: A value network that evaluates the actions taken by the actor, estimating the state-value V(s) or advantage A(s,a).
• Mechanism: The critic provides a lower-variance estimate of the expected reward, which is used to guide updates to the actor's policy parameters. This reduces the variance inherent in pure policy gradients.
• Examples: Advantage Actor-Critic (A2C), Asynchronous Advantage Actor-Critic (A3C), Soft Actor-Critic (SAC) (for continuous control), and Twin Delayed DDPG (TD3).

This distinction defines the relationship between the policy being evaluated/improved (the target policy) and the policy used to generate behavior and collect data (the behavior policy).

• On-Policy Learning: The agent learns about the same policy it is using to act. Data is collected under the current policy, and updates are made to improve that same policy. This ensures data relevance but can limit data efficiency. Examples include SARSA, A2C/A3C, and PPO.
• Off-Policy Learning: The agent can learn about a target policy (often the optimal policy) using data generated by a different behavior policy (e.g., an exploratory policy). This enables experience replay and learning from historical or expert data. Examples include Q-Learning, DQN, DDPG, and SAC.

A critical sub-category defined by the need to output actions in a continuous, real-valued domain (e.g., torque for a robot joint).

• Challenge: Value-based methods require an argmax over actions, which is intractable in continuous spaces.
• Primary Solutions:
  1. Policy-Based/Actor-Critic Methods: The actor network outputs the mean (and often variance) of a continuous distribution (e.g., Gaussian). PPO and SAC are dominant here.
  2. Value-Based with Optimization: The Deep Deterministic Policy Gradient (DDPG) algorithm uses an actor to propose actions, but the critic (a Q-network) is used to provide gradients to the actor via the deterministic policy gradient theorem.
  3. Normalized Advantage Functions (NAF): A variant of DQN that parameterizes the Q-function to make the max operation analytical.
• Key Libraries: Implementations are readily available in Stable-Baselines3, Ray RLlib, and Spinning Up.

A Markov Decision Process (MDP) is the foundational mathematical framework for modeling sequential decision-making in RL. It is formally defined by the tuple (S, A, P, R, γ), where:

• S is a set of states.
• A is a set of actions.
• P(s'|s, a) is the state transition probability function.
• R(s, a, s') is the reward function.
• γ is a discount factor (0 ≤ γ ≤ 1) that determines the present value of future rewards. The Markov property assumes the future state depends only on the current state and action, not the full history. All standard RL problems assume an underlying MDP.

Policy Gradient Methods are a class of RL algorithms that directly optimize a parameterized policy π_θ(a|s) using gradient ascent on the expected cumulative reward J(θ). Instead of learning a value function first, they adjust the policy parameters θ to increase the probability of high-reward trajectories.

Key aspects include:

• REINFORCE Algorithm: A Monte Carlo policy gradient method that uses entire episode returns.
• Advantage Function: Often used to reduce variance in gradient estimates (e.g., A(s,a) = Q(s,a) - V(s)).
• Direct Optimization: Well-suited for high-dimensional or continuous action spaces common in robotics control. These methods form the basis for advanced algorithms like PPO and are central to training visuomotor control policies.

Model-Based Reinforcement Learning is an approach where an agent learns an explicit model of the environment's dynamics—the transition function P(s'|s,a) and reward function R(s,a). This learned world model is then used for planning (e.g., via trajectory optimization or Model Predictive Control) or to generate synthetic experience to improve sample efficiency.

Contrast with model-free RL (like Q-Learning or Policy Gradients), which learns a policy or value function directly from interaction. Key advantages include:

• High Sample Efficiency: Can learn from fewer environmental interactions.
• Planning Capability: Enables lookahead and reasoning about consequences.
• Sim-to-Real Transfer: A model trained in simulation can be adapted for real-world deployment, a core technique in robotics.

Hierarchical Reinforcement Learning (HRL) decomposes a complex, long-horizon task into a hierarchy of simpler subtasks or skills (often called options or skills). This introduces temporal abstraction, where a high-level policy selects among low-level skills that execute for extended periods.

This is critical for robotics and embodied AI because it mirrors how humans solve problems:

• High-Level Controller: Decides on sub-goals (e.g., 'pick up the block').
• Low-Level Policy: Executes primitive actions to achieve the sub-goal (e.g., joint motor commands).
• Efficiency: Enables reuse of learned skills across different tasks and mitigates the credit assignment problem over long time horizons. Frameworks like the Options Framework formalize this approach.

Safe Reinforcement Learning is the subfield focused on developing algorithms that maximize performance while satisfying critical safety constraints, which is non-negotiable for physical systems like robots and autonomous vehicles. Unlike standard RL which only maximizes reward, safe RL formalizes constraints that must not be violated.

Common approaches include:

• Constrained MDPs (CMDPs): The agent must keep expected cumulative costs below a threshold.
• Lagrangian Methods: Convert the constrained optimization problem into an unconstrained one using dual variables.
• Risk-Sensitive Criteria: Optimize conditional value at risk (CVaR) instead of expected return. The goal is to ensure safe exploration during training and safe operation during deployment, preventing catastrophic failures in real-world control applications.

Sim-to-Real Transfer is the process of training an RL policy in a high-fidelity simulated environment and successfully deploying it on a physical robot. The core challenge is the reality gap—the discrepancy between simulation dynamics and real-world physics.

Techniques to bridge this gap include:

• Domain Randomization: Training with randomized simulation parameters (e.g., friction, masses, visual textures) to force the policy to be robust.
• System Identification: Calibrating the simulator's parameters using data from the real system.
• Adaptive Policies: Policies that can online-adapt to real-world observations.
• Physics Engines: Tools like MuJoCo, Isaac Gym, and PyBullet are industry standards for creating these training simulations. This paradigm is essential for scalable and safe robotics RL, as training directly on hardware is often prohibitively slow, expensive, or dangerous.