Reinforcement Learning from Pixels

The foundational architecture for RL from pixels. A Convolutional Neural Network (CNN) processes raw image frames to extract spatial features, which are then fed into a Q-Network to estimate action values.

• Frame Stacking: Typically stacks 4 consecutive frames as input to provide a sense of temporal dynamics (velocity, acceleration).
• Experience Replay: Stores past transitions (state, action, reward, next state) in a buffer to break temporal correlations and improve sample efficiency.
• Target Network: Uses a separate, slowly updated network to calculate stable Q-value targets, mitigating training instability.
• Example: The original 2015 DQN that achieved human-level performance on Atari 2600 games using only pixel inputs.

A parallelized, policy-gradient method designed for stability and efficiency with visual inputs. Multiple agent instances run in parallel on different environment copies, aggregating gradients to a central model.

• Actor-Critic Architecture: The Actor (policy) selects actions, while the Critic (value function) evaluates the state, reducing variance in policy updates.
• Advantage Function: Updates the policy based on the Advantage (A = Q(s,a) - V(s)), measuring how much better an action is than average.
• Asynchronous Updates: Eliminates the need for an experience replay buffer, allowing on-policy learning and often faster training on multi-core CPUs.
• Key Benefit: More stable than pure value-based methods (like DQN) for complex, continuous action spaces common in robotics.

The dominant on-policy algorithm for RL from pixels in robotics due to its robustness and ease of tuning. It optimizes a surrogate objective function that penalizes large policy updates.

• Clipped Objective: The core innovation. It constrains the policy update to a trust region by clipping the probability ratio, preventing destructively large steps.
• Generalized Advantage Estimation (GAE): Efficiently estimates the advantage function across multiple timesteps, balancing bias and variance.
• Multiple Epochs: Makes several optimization passes over a batch of data collected from the current policy, improving sample efficiency.
• Sim-to-Real Relevance: Its stability makes it a preferred choice for training in simulation, where long, stable training runs are required to learn complex pixel-to-action mappings.

An off-policy maximum entropy reinforcement learning algorithm. It excels in learning from pixels for continuous control tasks by maximizing both expected reward and policy entropy.

• Entropy Regularization: Encourages exploration by rewarding the policy for being stochastic (higher entropy), leading to more robust and diverse behaviors.
• Off-Policy Learning: Efficiently reuses past experience from a replay buffer, similar to DQN.
• Two Q-Functions & a Value Function: Uses twin Q-networks to mitigate overestimation bias and a separate state value function.
• Why it's used: Its sample efficiency and stability make it suitable for training in simulation where real-world data collection is impossible or expensive.

Applying random transformations to input images during training to improve visual robustness—a critical technique for bridging the sim-to-real visual gap.

• Common Augmentations: Random cropping, color jitter (brightness, contrast, saturation), Gaussian noise, and cutout.
• Mechanism: Forces the convolutional encoder to learn invariant features. A robot must recognize a red cube whether it's in bright light or shadow.
• DrAC (Data-regularized Actor-Critic): A framework that integrates augmentations directly into the RL objective, showing significant improvements in sim-to-real transfer for manipulation tasks.
• Analogy: Similar to augmentation in supervised computer vision, but applied within an active RL loop.

Techniques to learn better visual representations by adding self-supervised learning objectives alongside the primary RL reward signal.

• Contrastive Loss (e.g., CURL): Treats different augmentations of the same observation as positive pairs and different observations as negative pairs. This learns representations where visual semantics are preserved despite pixel-level changes.
• Auxiliary Tasks: Predicts environment properties like pixel control, reward prediction, or depth estimation. This provides a richer learning signal, especially in sparse-reward settings.
• Benefit for Sim-to-Real: By learning more general, semantically meaningful features, the policy becomes less sensitive to the low-level visual differences (textures, lighting) between simulation and reality.

The primary challenge for pixel-based policies is the domain shift between synthetic and real-world visuals. This includes discrepancies in:

• Textures, lighting, and colors
• Object shapes and rendering artifacts
• Sensor noise and distortion (e.g., lens blur, motion blur)
• Background clutter and distractors Unlike low-dimensional state observations, pixels contain many irrelevant features that can cause the policy to overfit to simulation-specific visual cues, leading to catastrophic failure in reality.

The most common method to bridge the visual gap. The policy is trained in simulation with a wide distribution of randomized visual parameters to force it to learn invariant features. Key randomized elements include:

• Textures: Applying random materials to objects and floors.
• Lighting: Varying number, position, color, and intensity of light sources.
• Camera Properties: Altering field of view, pose noise, and color gradients.
• Backgrounds: Using random images or 3D models. The goal is to make the policy rely on geometric and semantic shapes rather than exact colors or textures, improving generalization.

Techniques that actively transform images from one domain to another to align feature spaces.

• CycleGAN: An unsupervised method that learns to translate simulated images to look 'real' and vice versa without paired data, used to create more realistic training data.
• Pixel-Level Domain Adaptation: Training a perception module with a domain-adversarial loss (e.g., using a Gradient Reversal Layer) so the extracted features are indistinguishable between simulation and reality.
• Stylization: Using fast neural style transfer to re-render simulated frames in diverse real-world styles during training.

A robust design pattern is to use a two-stage network architecture:

1. Perception Module: A convolutional neural network (CNN) or Vision Transformer that processes raw pixels into a compact, task-relevant latent representation or state estimate.
2. Policy Network: A smaller multilayer perceptron that takes this latent state and outputs actions. This separation allows for targeted robustness techniques (like domain randomization on the perception module) and enables transfer of the perception module only if the robot's dynamics are similar.

A practical sequence for moving from simulation to physical hardware:

1. Train with Heavy Randomization: Policy learns from pixels under maximally varied visual conditions.
2. Validate in Simulation: Test policy in held-out, less-randomized simulation environments.
3. Zero-Shot Deployment: Attempt direct execution on the real robot. This often reveals unseen failures.
4. Fine-Tuning (if possible): Use limited real-world rollouts for on-policy adaptation. This may involve:
   • Fine-tuning the final layers of the perception network.
   • Training an adapter network that maps real features to the simulation-trained latent space.
   • Using real data to calibrate the simulation (system identification for visuals).

Common reasons pixel-based policies fail to transfer:

• Simulation Overfitting: Policy exploits unrealistic physical shortcuts (e.g., perfect friction) correlated with visual features.
• Partial Observability: Real-world camera views may be occluded or have different angles than simulation.
• Temporal Consistency: Real video has different frame rates and latency, disrupting policies sensitive to timing.
• Dynamic Objects: Simulation often lacks the variety and unpredictability of moving people or objects.
• Lighting Extremes: Policies trained in simulation often fail under harsh shadows, glare, or low-light conditions not modeled during randomization.

The Reality Gap is the fundamental discrepancy between the simulated environment used for training and the target real-world system. For pixel-based RL, this manifests as:

• Visual Domain Shift: Differences in lighting, textures, object appearance, and sensor noise between rendered and real camera images.
• Dynamics Mismatch: Inaccuracies in simulated physics (e.g., friction, motor models, contact dynamics) compared to physical hardware.
• Latency and Synchronization: Simulation typically assumes perfect, instantaneous sensing and actuation, unlike real-world sensor delays and communication jitter. This gap causes the Performance Drop observed when a simulation-trained policy is deployed on a physical robot.

Domain Randomization is a core sim-to-real technique that trains a policy under a wide spectrum of randomized simulation conditions to encourage robustness. For pixel-based RL, randomization targets include:

• Visual Parameters: Object textures, colors, lighting positions/intensities, camera noise, and background scenes.
• Physics Parameters: Object masses, friction coefficients, motor torque limits, and sensor latency ranges.
• Domain-Adversarial Training can be combined, where a network learns features invariant to these randomized domains. The goal is to create a Policy Robustness that generalizes to unseen real-world variations, enabling Zero-Shot Transfer without real-world fine-tuning.

Domain Adaptation refers to machine learning techniques that adapt a model trained on a source domain (simulation) to perform well on a different target domain (reality). Key approaches for visual RL include:

• Unsupervised Domain Adaptation: Uses Unpaired Data (sim and real images without correspondence). Techniques like CycleGAN learn a mapping to translate simulated pixels to a more photorealistic style.
• Paired Data methods require aligned sim-real image pairs, which are difficult to obtain but allow for supervised pixel-level translation.
• Feature-Level Adaptation: Learns domain-invariant representations in a latent feature space, often using adversarial losses, so the policy network processes features that look similar from both domains.

System Identification is the process of learning an accurate mathematical model of a physical robot's dynamics from observed data. System Calibration involves precisely measuring and adjusting sensor and actuator parameters. Together, they directly reduce the Dynamics Mismatch component of the reality gap.

• Process: The robot executes a series of motions (e.g., via a safe script); its sensor responses (joint positions, velocities) are recorded and used to fit the parameters of the simulation's physics engine.
• Impact on RL: A more physically accurate simulation means policies trained under it require less Robustness to dynamics errors, simplifying the transfer problem. It is often a prerequisite for Model Predictive Control (MPC) Transfer.

Residual Policy Learning is a hybrid control architecture designed to bridge the sim-to-real gap. A traditional, analytically derived controller (e.g., a PID or MPC) provides a baseline command. A neural network policy, trained via RL in simulation, learns to output a residual correction to this command.

• Advantage: The base controller ensures basic stability and safety, while the learned residual adapts to unmodeled dynamics and complexities. This decomposes the problem, making it easier for RL to learn the "delta" between the imperfect simulation and reality.
• Use Case: Commonly applied in robotic manipulation and legged locomotion, where the gap between simulated and real actuator dynamics is significant.

A World Model is a neural network that learns a compressed, latent representation of an environment and can predict future latent states. For pixel-based RL, this offers a powerful sim-to-real pathway:

• Process: A model (e.g., a Variational Autoencoder) is trained to encode high-dimensional pixels into a low-dimensional latent vector. A dynamics model is then learned in this latent space.
• Transfer Benefit: The policy is trained entirely within the learned latent dynamics model. If the world model can be adapted to real-world data (e.g., via fine-tuning the encoder), the policy may transfer more effectively, as it operates on abstract features rather than raw pixels susceptible to Visual Domain Shift.