Model-Free Reinforcement Learning

In model-free RL, the agent learns either a policy (a mapping from states/observations to actions) or a value function (estimating the future reward from a state or state-action pair) directly from experience. This contrasts with model-based methods that first learn a forward dynamics model. Common algorithms include:

• Policy Gradients (e.g., PPO): Directly optimize policy parameters.
• Value-Based Methods (e.g., DQN): Learn a Q-function to select actions.
• Actor-Critic Methods (e.g., SAC): Combine both a policy (actor) and a value estimator (critic).

The agent learns purely through interaction, receiving reward signals as feedback. It must navigate the fundamental exploration-exploitation tradeoff: balancing trying new actions to discover their effects (exploration) with choosing known rewarding actions (exploitation). This direct interaction is data-inefficient but avoids the complexity and potential inaccuracies of learning a world model, making it suitable for complex, high-dimensional domains like pixel-based visuomotor control.

A core tenet is the absence of an explicit, learnable dynamics model or transition function that predicts the next state given the current state and action. The agent does not plan by simulating futures within an internal model. Instead, it relies on sampled experience, often stored in an experience replay buffer, to update its policy or value estimates. This makes MFRL robust in environments where dynamics are difficult to model accurately but can be sample-inefficient.

MFRL algorithms, particularly when combined with deep neural networks (Deep RL), excel at processing raw, high-dimensional sensory inputs. For visuomotor control, this means learning policies that map directly from pixel-based visual observations to low-level motor commands. The policy network must learn to extract relevant state features (like object position) from pixels, a form of representation learning, without intermediate hand-engineered perception modules.

Model-free RL is the primary training framework for end-to-end visuomotor control. A neural network policy (e.g., a CNN or Vision Transformer) consumes image observations and outputs actions (e.g., joint torques). Through RL, this policy is trained to maximize task reward. This creates a tight perception-action cycle where visual features are learned specifically for the control task, often leading to more robust and adaptive behaviors than modular pipelines.

A major challenge is sample inefficiency; MFRL often requires millions of environment interactions to learn, which is prohibitive for physical robots. Techniques to improve stability and efficiency include:

• Experience Replay: Decorrelates training data by sampling from a buffer of past transitions.
• Target Networks: Uses slowly updated copies of value networks to stabilize training.
• Trust Region Methods (e.g., in PPO): Limit policy update sizes to prevent performance collapse.
• Entropy Regularization (e.g., in SAC): Encourages exploration by maximizing policy entropy.

MFRL has achieved superhuman performance in complex video games and simulated environments, serving as a benchmark for algorithm development. Notable examples include:

• Deep Q-Networks (DQN) mastering a wide range of Atari 2600 games from pixel inputs.
• AlphaStar (DeepMind) reaching Grandmaster level in StarCraft II, a game requiring long-term strategy and real-time decision-making.
• OpenAI Five defeating world champions in Dota 2, demonstrating coordination in a multi-agent setting with partially observable states.

In settings where the system dynamics are complex or unknown, MFRL optimizes control for manufacturing, resource management, and logistics. Applications include:

• Data center cooling: Agents learn to control fans, chillers, and vents to minimize energy consumption (PUE) while maintaining temperature constraints.
• Chemical process control: Optimizing reaction yields by adjusting temperatures, pressures, and flow rates in real-time.
• Warehouse robotics: Learning efficient picking and sorting policies that adapt to changing inventory layouts and order profiles.

MFRL is applied to dynamic allocation problems in telecommunications and computing. Agents learn to manage limited resources without a precise model of user demand or network conditions. Key use cases are:

• 5G/6G network slicing: Dynamically allocating bandwidth and compute resources across different service types (e.g., IoT, mobile broadband, ultra-reliable low-latency communications).
• Cloud computing: Autoscaling virtual machines and containers to meet application demand while minimizing cost and latency.
• Content delivery: Pre-fetching and caching digital content at edge nodes to reduce latency based on predicted user requests.

MFRL agents learn trading strategies by interacting with market simulators, optimizing for metrics like the Sharpe ratio or cumulative profit. They must handle high-dimensional, noisy data and manage risk under uncertainty. Applications include:

• Portfolio management: Dynamically allocating capital across a basket of assets.
• Optimal order execution: Breaking a large trade into smaller orders to minimize market impact and transaction costs.
• Market making: Providing liquidity by continuously quoting bid and ask prices, learning to manage inventory risk. These applications treat the market as a partially observable environment where the true state (e.g., other traders' intentions) is hidden.

The contrasting paradigm to model-free RL. Here, the agent learns or is provided with an explicit model of the environment's dynamics (a forward dynamics model). This model is then used for planning, simulating rollouts, or policy improvement via algorithms like Model Predictive Control (MPC).

• Key Difference: Uses an internal world model; model-free does not.
• Trade-off: Typically more sample-efficient but can suffer from model bias if the learned dynamics are inaccurate.
• Example: Using a physics simulator to predict the outcome of actions before executing them on a real robot.

A foundational class of model-free RL algorithms that directly optimize a parameterized policy. They work by estimating the gradient of expected reward with respect to the policy parameters and then ascending that gradient.

• Core Mechanism: Adjusts policy parameters in the direction that increases the probability of high-reward actions.
• Algorithms Include: REINFORCE, Proximal Policy Optimization (PPO), and Trust Region Policy Optimization (TRPO).
• Use in Robotics: Directly optimizes the visuomotor policy mapping pixels to torques.

A dominant architecture in modern model-free RL that combines two neural networks:

• The Actor: The policy itself, which selects actions.
• The Critic: A value function (e.g., state-value or advantage function) that evaluates the actions chosen by the actor.

The critic provides a lower-variance training signal to the actor than pure Monte Carlo returns. Algorithms like Soft Actor-Critic (SAC) and TD3 are actor-critic methods essential for learning continuous control policies in robotics.

A critical technique for improving the data efficiency and stability of model-free RL, especially in off-policy algorithms. The agent stores its experiences (state, action, reward, next state) in a replay buffer and later samples random mini-batches from it for training.

• Breaks Temporal Correlations: Sequential experiences are highly correlated; random sampling decorrelates them.
• Enables Off-Policy Learning: Allows reuse of past data collected by older policies.
• Example: A robotic arm learns from thousands of past grasping attempts, not just its most recent trials.

The fundamental dilemma every model-free RL agent must solve. The agent must balance exploring new actions to discover potentially higher rewards with exploiting known actions that yield good reward.

• Pure Exploration: Randomly trying actions; inefficient.
• Pure Exploitation: Sticking to known good actions; may miss optimal strategies.
• Model-Free Solutions: Algorithms use mechanisms like entropy regularization (in SAC) or epsilon-greedy policies to manage this trade-off automatically during training.

The formal mathematical framework for the problem model-free RL often solves in robotics. In a POMDP, the agent does not have direct access to the true state of the environment. Instead, it receives only partial or noisy observations (e.g., a single camera image).

• Reality for Visuomotor Policies: A robot's camera provides a partial observation, not the full kinematic state.
• Implication: The policy must learn to infer relevant state information from a history of observations.
• Connection: Model-free RL algorithms like recurrent PPO are used to train policies in POMDPs.