Policy Network

The observation space defines what sensory data the policy network receives. For dexterous manipulation, this is typically multimodal and high-dimensional.

• Proprioceptive State: Joint angles, velocities, and motor torques.
• Exteroceptive Sensing: Visual data from wrist-mounted cameras or GelSight tactile sensors.
• Task Context: The goal state, such as a target object 6D pose. Effective policy networks fuse these streams into a unified state representation for decision-making.

The action space defines the control outputs of the policy. For fine motor control, this requires careful design.

• Joint Space vs. Cartesian Space: Commands can be target joint positions/torques or desired end-effector impedance control parameters.
• Action Tokenization: Continuous movements can be discretized into tokens for sequence modeling.
• Dimensionality: Must match the robot's degrees of freedom, which can be high for multi-fingered hands, often requiring underactuation modeling.

The core neural network design balances expressivity with training stability.

• Multilayer Perceptrons (MLPs): Common for processing fused state vectors.
• Recurrent Layers: LSTMs or GRUs can provide temporal memory, crucial for contact sequences in in-hand manipulation.
• Convolutional Encoders: Process raw pixel data from visual observations.
• Attention Mechanisms: Allow the policy to focus on relevant visual features or past states. Architectures are often hybrid, combining these elements.

Policies can output a single action (deterministic) or a distribution over actions (stochastic).

• Stochastic Policies: Output parameters (e.g., mean and variance) of a probability distribution (e.g., Gaussian). Essential for exploration during reinforcement learning and for handling task uncertainty.
• Deterministic Policies: Output a single action value. Used in imitation learning or after sufficient exploration. In dexterous tasks, stochastic policies help discover robust regrasping strategies.

Policy networks are trained via specific algorithms that define the learning signal.

• Reinforcement Learning (RL): Trained to maximize cumulative reward, using algorithms like PPO or SAC. Requires careful reward shaping for contact-rich tasks.
• Imitation Learning (IL): Trained via behavior cloning to mimic expert demonstrations, bypassing reward design but risking distributional shift.
• Hybrid Approaches: Combine IL for initialization with RL for fine-tuning, a common strategy to overcome the sim-to-real gap.

For closed-loop robotic control, policy networks must execute with extremely low latency.

• Inference Speed: Must often run at >100 Hz to provide stable impedance or admittance control. This demands optimized models and hardware acceleration.
• Model Compression: Techniques like quantization and pruning are applied post-training to reduce computational load.
• Specialized Hardware: Deployment on GPUs, NPUs, or embedded systems is critical for real-time robotic perception and action.

A visuomotor policy network takes raw camera images as input and outputs low-level joint torque commands for a robotic hand. This end-to-end approach is used for complex tasks like finger gaiting (walking fingers to rotate an object) or precision regrasping. These networks are typically trained via Deep Reinforcement Learning (DRL) in simulation with domain randomization to bridge the sim-to-real gap. Real-world deployment requires robust proprioceptive and exteroceptive sensing fusion.

These networks take both visual observations and a natural language instruction (e.g., "put the blue block on the red cup") to generate actions. The policy must perform visual grounding to associate words with objects and then plan the sequence of non-prehensile (e.g., pushing) and prehensile (grasping) actions. This is a core capability for human-robot interaction in unstructured environments, moving beyond single, pre-programmed skills.

For contact-rich tasks like inserting a peg or following a contour, a policy network processes high-dimensional tactile sensor data (e.g., from a GelSight sensor) in real-time. Using a Recurrent Neural Network (RNN) architecture like an LSTM or GRU allows the policy to maintain a memory of contact dynamics. The network outputs corrective velocity or force commands, implementing a form of learned impedance or admittance control to achieve robust manipulation despite uncertainty.

Complex tasks like "make coffee" are decomposed using a hierarchical policy architecture. A high-level policy (or planner) outputs sub-goals (e.g., "grasp mug," "navigate to machine") at a slow timescale. A low-level policy network then executes each sub-goal, generating the specific joint commands. This separation simplifies learning and improves sample efficiency. The high-level policy can be a learned network or a symbolic task and motion planning module.

A residual policy network learns to adapt and improve upon a base controller. For example, a traditional inverse kinematics solver provides nominal motion, but a learned residual policy observes state error and adds fine-tuning adjustments. This is often trained via imitation learning from human demonstrations, allowing the system to compensate for dynamic effects and model inaccuracies that are hard to capture analytically. It combines the safety of model-based control with the adaptability of learning.

Here, the "policy" is the online optimization performed by an MPC controller. A neural network is used as the learned dynamics model within the MPC loop, predicting future states given current states and proposed actions. The optimizer then finds the action sequence that minimizes cost (e.g., distance to goal, energy use). This provides a powerful framework for contact-implicit trajectory optimization and reactive control, as the plan is re-computed at every timestep based on the latest observations.

A specialized policy network that directly maps high-dimensional visual observations (e.g., camera images) to low-level motor commands. In dexterous manipulation, this enables end-to-end learning of tasks like in-hand reorientation or peg insertion without explicit state estimation.

• Key Feature: Processes raw pixels to output joint torques or velocities.
• Architecture: Often uses a CNN backbone for visual feature extraction followed by an MLP for control.
• Challenge: Requires massive, diverse training data to achieve robustness to visual variations.

An advanced online optimization-based control method that uses a dynamic model to predict future system states and iteratively solves for an optimal sequence of control inputs. Unlike a learned policy network, MPC re-plans at each time step.

• Contrast with Policy Networks: MPC is model-based and explicitly optimizes; policy networks are model-free and encode a reactive mapping.
• Use in Dexterous Manipulation: Often used for high-frequency, contact-rich tasks where accurate dynamics models are available, sometimes in a hybrid approach where a policy network provides a warm start for the MPC solver.

A paradigm for training policy networks using demonstrations from an expert (human or algorithm) rather than through trial-and-error reinforcement learning. Behavior Cloning is a primary technique.

• Process: The policy network learns to mimic the expert's action given an observation.
• Application: Common for bootstrapping dexterous manipulation policies from human teleoperation or motion-captured data.
• Limitation: Can suffer from distributional shift; small errors compound when the policy encounters states not seen in the demonstration data.

A mathematical framework for representing reproducible, smooth trajectories that can be spatially and temporally scaled. While not a neural network policy itself, DMPs are often used as a parameterized policy representation or as building blocks within a larger policy architecture.

• Key Property: Robustness to perturbations and ease of adaptation to new goals.
• Role in Policy Networks: A policy network might learn to output the goal parameters or forcing function weights for a DMP, which then generates the precise motor command sequence for a manipulation skill.

A foundational reinforcement learning framework where the policy network (the actor) is trained in tandem with a value network (the critic). The critic evaluates the actor's actions, providing a learning signal to improve the policy.

• Actor (Policy Network): Maps state to action probability distribution.
• Critic (Value Network): Estimates the expected cumulative reward from a given state.
• Significance: Most modern policy networks for dexterous manipulation (e.g., using PPO, SAC, DDPG) are trained within an actor-critic setup, enabling stable learning from sparse rewards.

A learned or engineered compact representation of an environment that enables prediction of future states. In advanced RL for manipulation, a policy network may be trained inside a learned world model (e.g., using Dreamer).

• Function: Compresses high-dimensional observations (like images) into a latent state that captures essential dynamics.
• Interaction with Policy: The policy network operates on this latent state, and its proposed actions are evaluated by rolling out imagined futures in the world model, drastically improving sample efficiency compared to training solely in the real world.