Reinforcement Learning (RL)

The agent is the autonomous decision-maker that learns a policy. Its core functions are:

• Perception: Receives an observation of the environment's state.
• Decision-Making: Selects an action based on its current policy.
• Learning: Updates its policy based on the reward received and the resulting new state.

In robotics, the agent is the software controller for the robot, deciding motor commands.

The environment is the world with which the agent interacts. It encompasses everything outside the agent. Key characteristics include:

• State (s): A complete description of the world at a timestep.
• Dynamics/Model: The rules (often probabilistic) that determine the next state given the current state and the agent's action.
• In robotics, this can be the real physical world or a physics simulation used for training.

A critical distinction in embodied systems:

• State (s): The true, full configuration of the environment (e.g., exact positions, velocities). Often not fully knowable.
• Observation (o): The partial or noisy sensory data the agent actually receives (e.g., camera image, LiDAR point cloud, joint encoder readings).
• This leads to the Partially Observable Markov Decision Process (POMDP) framework, which is standard for real-world robotics.

The set of all valid actions the agent can take. It defines the agent's degree of control.

• Discrete: A finite set of choices (e.g., {move_left, move_right, jump}).
• Continuous: An infinite set within a bounded range (e.g., torque applied to a joint: [-5 Nm, 5 Nm]).
• Robotic control is predominantly continuous and high-dimensional (e.g., controlling 7 joints on an arm).

The policy is the agent's strategy; it maps states (or observations) to actions. It is the function being learned.

• Deterministic Policy: π(s) = a. Directly outputs an action.
• Stochastic Policy: π(a|s). Outputs a probability distribution over actions, crucial for exploration.
• In deep RL, the policy is typically parameterized by a neural network (e.g., a Policy Network).

The reward function R(s, a, s') provides a scalar feedback signal to the agent. It is the primary guide for learning.

• Design Challenge: The reward function must be carefully engineered to incentivize the desired long-term behavior. A poor reward can lead to unintended, reward-hacking policies.
• Sparse vs. Dense: Sparse rewards (e.g., +1 for task success, 0 otherwise) are hard to learn from. Reward shaping adds intermediate rewards to guide learning.

The Markov Decision Process is the foundational mathematical framework for modeling sequential decision-making. An MDP is formally defined by the tuple (S, A, P, R, γ), where:

• S is a set of states.
• A is a set of actions.
• P(s'|s,a) is the state transition probability function.
• R(s,a,s') is the reward function.
• γ is a discount factor (0 ≤ γ ≤ 1) that determines the present value of future rewards. The Markov property assumes the future state depends only on the current state and action, not the full history. All standard RL problems are formulated as MDPs or their extensions.

Q-Learning is a foundational, model-free, off-policy algorithm for learning the optimal action-value function, denoted as Q*(s,a). This function represents the maximum expected cumulative reward achievable from a state-action pair. The algorithm updates its estimates using the Bellman optimality equation via Temporal Difference (TD) learning: Q(s,a) ← Q(s,a) + α [ r + γ max_a' Q(s',a') - Q(s,a) ] where α is the learning rate. Because it learns the value of the optimal policy independently of the actions taken (off-policy), it is highly flexible. Its simplicity makes it a benchmark, but it struggles with large, continuous state spaces without function approximation.

Policy Gradient Methods are a class of algorithms that directly optimize the parameters θ of a policy π_θ(a|s), which is a probability distribution over actions given a state. Instead of learning a value function first, they ascend the gradient of the expected return J(θ). The fundamental REINFORCE algorithm uses the gradient estimator: ∇_θ J(θ) ≈ E[ G_t ∇_θ log π_θ(A_t|S_t) ] where G_t is the return from time t. This approach is inherently on-policy and excels in continuous action spaces (e.g., robot joint torques) where discrete action selection is infeasible. Advanced variants like PPO and TRPO add constraints to ensure stable, monotonic improvement.

The Actor-Critic Architecture combines the strengths of value-based and policy-based methods. It consists of two components:

• The Actor: A policy network π_θ(a|s) that selects actions.
• The Critic: A value network V_φ(s) or Q_φ(s,a) that evaluates the state or state-action pair chosen by the actor. The critic provides a lower-variance estimate of the advantage (how much better an action is than average) to guide the actor's policy updates. This reduces the high variance of pure policy gradients (like REINFORCE) and accelerates learning. Algorithms like A3C, DDPG, and SAC are all built on this architecture, making it the dominant paradigm for modern deep RL.

The Exploration-Exploitation Tradeoff is the fundamental dilemma an RL agent must resolve: whether to exploit known actions that yield good reward or explore new actions to discover potentially better long-term strategies. Poor exploration can lead to suboptimal policy convergence. Common strategies include:

• ε-greedy: Select a random action with probability ε, else the greedy action.
• Softmax (Boltzmann): Select actions proportionally to their estimated value.
• Upper Confidence Bound (UCB): Prefer actions with high uncertainty in their value estimate.
• Intrinsic Motivation: Generate internal reward for visiting novel or uncertain states. Balancing this tradeoff is critical for sample efficiency and final performance.

Model-Based Reinforcement Learning involves learning or being given a model of the environment's dynamics—a function that predicts the next state and reward given the current state and action. This model can then be used for planning (e.g., via Monte Carlo Tree Search or trajectory optimization) to select actions without direct trial-and-error in the real environment. The primary advantage is dramatically improved sample efficiency, as data can be used to improve the model, which then generates unlimited simulated experience. The core challenge is model bias; inaccuracies in the learned model can compound during planning, leading to poor performance. It is often combined with model-free components for robustness.