Reinforcement Learning (RL)

The agent is the autonomous decision-maker that learns a policy—a mapping from perceived environmental states to actions—to maximize cumulative reward. It comprises the learning algorithm (e.g., a neural network) and the policy being optimized. The agent's core functions are perception (observing states), decision-making (selecting actions via its policy), and learning (updating its policy based on reward feedback).

The environment is everything the agent interacts with outside of itself. It is a dynamic system that, given an action from the agent, transitions to a new state and emits a scalar reward. The environment can be:

• Physical: A real-world robot arm or game.
• Simulated: A physics-based digital model used for safe, parallelized training. Its dynamics are defined by transition probabilities P(s' | s, a) and a reward function R(s, a, s').

A state (s) is a complete description of the environment at a timestep. An observation (o) is a partial or noisy perception of that state received by the agent. This distinction is critical in Partially Observable Markov Decision Processes (POMDPs), common in robotics where sensors provide limited data (e.g., a camera image instead of perfect object positions). The agent's policy is typically conditioned on its observation history.

An action (a) is a decision or control signal output by the agent that influences the environment. Action spaces are categorized as:

• Discrete: A finite set of choices (e.g., move left/right).
• Continuous: A real-valued vector (e.g., joint torque values).
• Hybrid: A mix of both. Algorithms are specialized for these spaces; Deep Q-Networks (DQN) excel in discrete spaces, while Proximal Policy Optimization (PPO) or Soft Actor-Critic (SAC) are designed for continuous control.

The reward function R(s, a, s') provides a scalar feedback signal that defines the agent's goal. It is the primary mechanism for credit assignment, quantifying the desirability of state-action transitions. A well-designed reward function is sparse or dense:

• Sparse rewards (e.g., +1 for task completion, 0 otherwise) are simple but make exploration difficult.
• Dense rewards provide incremental feedback (e.g., distance to a goal) to guide learning but risk reward hacking, where the agent exploits loopholes without solving the intended task.

The policy (π) is the agent's strategy, formally a probability distribution over actions given a state (π(a|s)). It is the function being learned.

• Deterministic Policy: Maps directly to a single action (a = μ(s)).
• Stochastic Policy: Specifies a probability for each action, crucial for exploration. Policies are often parameterized by neural networks (π_θ). The learning objective is to find parameters θ that maximize the expected return (sum of discounted future rewards).

A value function estimates the expected long-term return from a given state or state-action pair, serving as a guide for the policy.

• State-Value Function V(s): Expected return starting from state s.
• Action-Value Function Q(s, a): Expected return after taking action a in state s. These functions satisfy the Bellman equation, a recursive relationship that is the foundation for algorithms like Temporal Difference (TD) Learning and Q-Learning. The advantage function A(s, a) = Q(s, a) - V(s) measures the relative benefit of a specific action.

In Model-Based Reinforcement Learning, the agent learns or is given a model of the environment. This is an internal representation of the environment's dynamics, comprising:

• A transition model predicting the next state: s' ~ P_φ(s' | s, a).
• A reward model predicting the immediate reward: r ~ R_ψ(r | s, a). The agent can use this model for planning (e.g., via Monte Carlo Tree Search) or to generate synthetic experience for policy training, which can dramatically improve sample efficiency compared to model-free methods.

The Markov Decision Process (MDP) is the foundational mathematical framework for modeling sequential decision-making in reinforcement learning. It is defined by the tuple (S, A, P, R, γ), where:

• S is a set of states.
• A is a set of actions.
• P(s' | s, a) is the state transition probability function.
• R(s, a, s') is the reward function.
• γ (gamma) is the discount factor (0 ≤ γ ≤ 1). The core assumption is the Markov property: the future state depends only on the current state and action, not the full history. Most RL problems are formalized as MDPs or their partially observable extensions (POMDPs).

The exploration-exploitation tradeoff is the fundamental dilemma an RL agent must solve. Exploitation means selecting actions known from past experience to yield high reward. Exploration means trying new or less certain actions to gather information and potentially discover higher-reward strategies. An optimal policy must balance this tradeoff. Common strategies include:

• ε-greedy: With probability ε, take a random action (explore); otherwise, take the best-known action (exploit).
• Upper Confidence Bound (UCB): Select actions based on a confidence interval that favors less-tried options.
• Thompson Sampling: A Bayesian approach that samples from a posterior distribution over action values. Failure to explore sufficiently can lead to suboptimal policies.

This distinction categorizes RL algorithms based on whether they learn or use an explicit model of the environment.

Model-Based RL: The agent learns (or is given) an explicit model of the environment's dynamics (P) and reward function (R). This model can be used for planning (e.g., via Monte Carlo Tree Search) to simulate future trajectories and choose actions. It can be highly sample-efficient but is sensitive to model inaccuracies.

Model-Free RL: The agent learns a policy or value function directly from experience, without constructing an explicit world model. Algorithms like Q-Learning, Policy Gradient, and Actor-Critic are model-free. They are often more robust to complex environments but can be less sample-efficient. Hybrid approaches also exist.

This distinction defines the relationship between the policy being evaluated/improved and the policy used to generate data.

On-Policy Learning: The agent learns the value of the policy it is currently executing. The behavior policy (used to act) and the target policy (being improved) are identical. Examples include SARSA and the vanilla REINFORCE algorithm. Updates require fresh data from the current policy.

Off-Policy Learning: The agent learns about a target policy (often the optimal policy) while following a different behavior policy (e.g., an exploratory one). This enables learning from historical data, demonstrations, or the actions of other agents. Q-Learning and DDPG are classic off-policy algorithms, leveraging a replay buffer of past experiences.

Value-Based Methods focus on learning a value function—either the state-value function V(s) or the action-value function Q(s, a)—that estimates expected future return. The optimal policy is then derived by selecting actions that maximize this value (e.g., argmax_a Q(s, a)).

Core Algorithms:

• Q-Learning: A model-free, off-policy algorithm that directly learns the optimal Q-function via the Bellman optimality equation: Q(s,a) ← Q(s,a) + α [r + γ max_a' Q(s',a') - Q(s,a)].
• Deep Q-Network (DQN): Uses a deep neural network to approximate Q(s,a) and introduced key stabilizers like experience replay and target networks. These methods are well-suited for discrete action spaces but require extra techniques (like DDPG) for continuous control.

Policy-Based Methods directly parameterize and optimize the policy π(a | s; θ) without relying on a separate value function. They search the policy space to maximize the expected cumulative reward J(θ).

Advantages: Naturally handle continuous action spaces, can learn stochastic policies, and often have better convergence properties.

Core Algorithm: Policy Gradient Theorem provides the gradient of the objective. The REINFORCE algorithm is a simple Monte Carlo policy gradient method.

Modern Algorithms:

• Proximal Policy Optimization (PPO): Uses a clipped surrogate objective to enable stable, large updates.
• Trust Region Policy Optimization (TRPO): Constrains policy updates based on KL divergence for monotonic improvement. These are typically on-policy, requiring data from the current policy, though actor-critic hybrids improve efficiency.