Model-Free RL

Model-free algorithms learn either a policy (a mapping from states to actions) or a value function (an estimate of future reward) directly from sampled experience. They do not learn or utilize an explicit model of the environment's transition dynamics T(s'|s,a) or reward function R(s,a). This is achieved through methods like:

• Policy Gradients: Directly optimizing policy parameters via gradient ascent on expected reward.
• Temporal-Difference (TD) Learning: Updating value estimates based on the difference between predicted and observed outcomes.
• Q-Learning: Iteratively approximating the optimal action-value function using the Bellman equation.

Learning is driven entirely by trajectories—sequences of states, actions, and rewards—collected from actual or simulated interaction. Parameter updates are performed using these empirical samples rather than calculations over a known model. This characteristic:

• Enables application to complex, unknown environments where dynamics are difficult to model analytically.
• Introduces variance and noise into the learning process, as updates rely on specific sampled outcomes.
• Defines the core challenge of sample efficiency, making techniques like experience replay and parallelized environment sampling critical for practical training.

Without a model to simulate outcomes, the agent must actively explore the environment to discover rewarding states and actions, while also exploiting its current knowledge to maximize reward. Model-free algorithms implement explicit mechanisms to manage this trade-off:

• ε-greedy: Selects a random action with probability ε, otherwise the greedy action.
• Softmax (Boltzmann) exploration: Selects actions based on a probability distribution derived from estimated values.
• Entropy regularization: Encourages stochastic policies to maintain a baseline level of exploration, as used in Soft Actor-Critic (SAC). Failure to explore sufficiently can lead to suboptimal policy convergence.

A fundamental division in model-free RL is how the agent uses collected experience.

• On-Policy Learning (e.g., PPO, A2C): The agent learns the value of the policy it is currently executing. Data is collected under the latest policy and then used for an update, after which the data is discarded. This ensures stability but can be less sample efficient.
• Off-Policy Learning (e.g., Q-Learning, DDPG, SAC): The agent learns about a target policy (often the optimal policy) using data generated by a different behavior policy (e.g., an exploratory policy). This enables learning from historical data stored in a replay buffer, dramatically improving sample efficiency.

Model-free RL is the dominant approach for training policies for physical robotic tasks like manipulation and locomotion. Its suitability stems from:

• Complex, Hard-to-Model Dynamics: Real-world physics involving friction, deformation, and contact are exceptionally difficult to model with perfect fidelity.
• High-Dimensional Continuous Action Spaces: Algorithms like PPO, SAC, and DDPG are designed to output continuous torque commands for robot joints.
• Sim-to-Real Transfer: Policies are typically trained in physics-based simulation (a cheap, fast, parallelizable source of samples) and then transferred to real hardware using techniques like domain randomization, leveraging the model-free agent's ability to learn robust behaviors without an exact world model.

The choice between model-free and model-based RL involves key engineering trade-offs:

• Sample Efficiency: Model-based methods are typically more sample-efficient as they can use a learned model for planning and internal simulation. Model-free methods often require orders of magnitude more environment interactions.
• Asymptotic Performance & Robustness: Model-free methods often achieve higher final performance on complex tasks because they are not limited by model bias or inaccuracies. They learn policies that can be robust to the imperfections a model would introduce.
• Computational Cost: Model-free updates are generally computationally cheaper per sample. Model-based methods incur the cost of learning and repeatedly querying a dynamics model.
• Complexity of Implementation: Modern model-free algorithms (e.g., PPO) are often considered more straightforward to implement and tune reliably than hybrid model-based approaches.

Q-Learning is a foundational, off-policy, value-based algorithm. It learns an action-value function (Q-function) that estimates the expected return for taking a given action in a state and following the optimal policy thereafter. The core update uses the Bellman optimality equation. Deep Q-Networks (DQN) scale this to high-dimensional state spaces (like images) by using a neural network to approximate the Q-function, stabilized with techniques like a replay buffer and target networks. It is best suited for discrete action spaces.

• Type: Value-based, Off-policy
• Key Innovation: Function approximation for high-dimensional states
• Primary Use: Games (Atari, Go), discrete decision-making

Policy Gradient algorithms are a class of on-policy methods that directly optimize a parameterized policy function. Instead of learning a value function first, they adjust the policy parameters in the direction that increases the expected cumulative reward, as estimated by sampling trajectories. The REINFORCE algorithm is a simple Monte Carlo policy gradient. These methods naturally handle continuous action spaces and stochastic policies but can suffer from high variance and poor sample efficiency.

• Type: Policy-based, On-policy
• Key Feature: Direct policy optimization, supports continuous actions
• Challenge: High variance in gradient estimates

Actor-Critic methods combine the strengths of value-based and policy-based approaches. The actor is a policy network that selects actions. The critic is a value function network (e.g., state-value) that evaluates the actions taken by the actor. The critic's evaluation provides a lower-variance baseline for updating the actor's policy. This hybrid approach typically offers more stable and sample-efficient learning than pure policy gradients.

• Type: Hybrid (Actor + Critic)
• Mechanism: Actor proposes actions, Critic provides feedback
• Benefit: Reduced variance compared to pure policy gradients

Proximal Policy Optimization (PPO) is a state-of-the-art, on-policy actor-critic algorithm designed for stability and ease of use. Its core innovation is a clipped surrogate objective that prevents excessively large policy updates, which could lead to performance collapse. PPO approximates the benefits of more complex constrained optimization methods (like TRPO) but with simpler first-order optimization. It has become a default choice for continuous control tasks in simulation and robotics due to its reliable performance.

• Type: On-policy Actor-Critic
• Key Mechanism: Clipped objective for stable updates
• Common Application: Robotic sim-to-real training, continuous control

Soft Actor-Critic (SAC) is an off-policy actor-critic algorithm that incorporates maximum entropy principles. It aims to maximize both expected reward and the entropy (randomness) of the policy. This encourages more robust exploration and leads to the agent learning multiple ways to succeed at a task. SAC is specifically designed for continuous action spaces and is known for its sample efficiency and stability, making it a leading algorithm for real-world robotic learning where data is expensive.

• Type: Off-policy Actor-Critic (Maximum Entropy)
• Key Principle: Maximizes reward and policy entropy
• Strength: Sample-efficient, robust exploration for robotics

Deep Deterministic Policy Gradient (DDPG) is an off-policy actor-critic algorithm designed for continuous action spaces. It combines insights from DQN (replay buffer, target networks) with a deterministic policy gradient. The actor outputs a deterministic action, and the critic learns a Q-function to evaluate state-action pairs. While powerful, it can be less stable than PPO or SAC and is sensitive to hyperparameters. Its twin variant, Twin Delayed DDPG (TD3), addresses overestimation bias in the Q-function for improved performance.

• Type: Off-policy Actor-Critic
• Design: Adapts DQN for continuous actions via deterministic policy
• Note: Often superseded by TD3 for improved stability

A foundational class of model-free algorithms that update value estimates by blending current predictions with observed rewards and subsequent predictions. Key mechanisms:

• Bootstrapping: Updates estimates using other estimates, enabling learning from incomplete sequences.
• Online Learning: Can learn after every time step without waiting for an episode's end.
• Core to Q-Learning & SARSA: Forms the update rule for these seminal algorithms.

A direct approach to model-free RL that optimizes a parameterized policy function by ascending the gradient of expected reward. Characteristics:

• Direct Policy Optimization: Adjusts policy parameters to increase the probability of high-reward actions.
• Suitable for Continuous Actions: Naturally handles high-dimensional, continuous action spaces.
• Examples: REINFORCE, Proximal Policy Optimization (PPO), and Deep Deterministic Policy Gradient (DDPG) (an actor-critic hybrid).

A hybrid model-free framework combining two networks: an Actor that selects actions and a Critic that evaluates them. How it works:

• The Critic (value function) reduces variance in policy updates by providing a baseline.
• Enables more stable and sample-efficient learning than pure policy gradients.
• Algorithms like A3C, SAC, and TD3 are built on this architecture, blending value-based and policy-based ideas.

A critical technique for sample efficiency in off-policy, model-free algorithms like DQN and DDPG. Function:

• Stores past experiences (state, action, reward, next state, done) in a finite buffer.
• Batches are randomly sampled (minibatch update) for training, which:
  • Breaks harmful temporal correlations in sequential data.
  • Reuses experiences, dramatically improving data efficiency.
• Enables off-policy learning by decoupling behavior from training data.

The fundamental dilemma in model-free RL, where an agent lacks a model to predict outcomes and must discover them. Strategies include:

• ε-greedy: Randomly explores with probability ε, otherwise exploits the best-known action.
• Noise Injection: Adds structured noise (e.g., Ornstein-Uhlenbeck) to actions for exploration in continuous control.
• Intrinsic Motivation: Encourages exploration of novel or uncertain states via bonus rewards.
• Entropy Regularization: Used in algorithms like SAC to maximize policy entropy, promoting diverse action selection.

A core distinction in model-free RL based on the relationship between the policy being evaluated and the policy generating data.

• On-Policy (e.g., SARSA, PPO): Learns the value of the policy currently being used for exploration. Data is fresh but correlated; the policy must be updated with its own actions.
• Off-Policy (e.g., Q-Learning, DDPG, SAC): Learns the value of an optimal policy (target) from data generated by a different policy (behavior). Enables learning from past data, demonstrations, or more exploratory policies, greatly enhancing data utility.