Policy Network

Policy networks are categorized by their output type. A stochastic policy outputs a probability distribution over actions (e.g., a categorical distribution for discrete actions, a Gaussian for continuous actions), enabling exploration. A deterministic policy maps directly to a single action value, useful for environments where the optimal action is unambiguous. Most on-policy algorithms (e.g., PPO) use stochastic policies, while off-policy algorithms (e.g., DDPG) often use deterministic ones.

The neural network architecture varies by domain:

• Multilayer Perceptron (MLP): For fully observed state vectors (e.g., joint angles, object positions).
• Convolutional Neural Network (CNN): Processes high-dimensional visual observations (pixels) for visuomotor control.
• Recurrent Neural Network (RNN) / Transformer: Handles partial observability or long-horizon tasks by maintaining internal state across time steps.
• Conditional Networks: Goal-conditioned policies take an additional goal vector as input. Skill-conditioned policies are primitives activated by a higher-level controller.

Policies are trained via several core methodologies:

• Reinforcement Learning (RL): The policy is optimized to maximize cumulative reward via algorithms like PPO, SAC, or DDPG. This involves trial-and-error interaction, often using a value function or critic to estimate action quality.
• Imitation Learning (IL): The policy learns to mimic expert demonstrations. Behavioral Cloning is direct supervised learning on state-action pairs. Inverse Reinforcement Learning infers the reward function behind the demonstrations first.
• Combined Approaches: RL fine-tuned from IL uses imitation to initialize a policy, which is then refined with RL for robustness and performance beyond the demonstration data.

This foundational theorem enables gradient-based optimization of stochastic policy parameters θ. It states that the gradient of the expected return J(θ) can be estimated from trajectories sampled by following the policy: ∇_θ J(θ) ≈ E[ Σ ∇_θ log π_θ(a|s) * A(s,a) ], where π_θ(a|s) is the policy's probability of action a in state s, and A(s,a) is the advantage function estimating how much better that action is than average. This allows the policy to be updated in the direction that increases the probability of high-advantage actions.

In physical systems, policy networks are the 'brain' converting perception to action.

• Visuomotor Policies: Map camera images directly to joint torques or end-effector velocities (e.g., CNN → MLP).
• Observation Space: Includes proprioception (joint states), exteroception (images, depth), and sometimes language instructions.
• Action Space: Can be continuous (joint velocities, torques) or discrete (high-level skills). Output is often sent to a low-level PID controller.
• Deployment Challenges: Require robustness to sensor noise, real-time inference (<100ms), and safety via action masking or safety critics.

Modern architectures address limitations of simple feedforward policies:

• Diffusion Policy: Models action sequences as a conditional denoising diffusion process, excelling at capturing multi-modal action distributions.
• Decision Transformer: Models RL as sequence modeling, autoregressively predicting actions conditioned on past states, actions, and a desired return-to-go token.
• Hierarchical Policy: A two-level system where a high-level meta-policy selects skill primitives (sub-policies) every N steps, which then execute low-level actions, enabling long-horizon reasoning.

A visuomotor policy network directly maps raw camera images to joint torque commands. This end-to-end approach bypasses explicit state estimation.

• Example: A robot learns to pick up diverse objects by observing its own gripper camera.
• Architecture: Typically a CNN processes images, with an MLP head outputting continuous motor commands.
• Training: Often trained via imitation learning from human demonstrations or reinforcement learning in simulation.

A goal-conditioned policy network uses natural language instructions to guide a mobile robot. The policy must ground linguistic concepts (e.g., 'kitchen') to visual features and plan a path.

• Example: 'Bring me the coffee mug from the counter.' The policy fuses CLIP-like visual-language embeddings with LiDAR scans to output velocity commands.
• Output: Continuous values for linear and angular velocity (Twist commands).
• Challenge: Requires robust cross-modal alignment between words, objects, and spatial layouts.

A hierarchical policy decomposes complex tasks. A high-level network selects skill primitives (e.g., 'open drawer', 'grasp'), while low-level networks execute the detailed movements.

• Example: Making a cup of coffee involves sequencing skills like move_to_kettle, grasp_handle, pour.
• High-Level Policy: Often a transformer that outputs a discrete skill token every few seconds.
• Low-Level Policy: A dedicated network (e.g., Diffusion Policy) that generates the precise joint motions for the selected skill.

A residual policy network acts on top of a classical controller. It learns to output corrective actions to adapt a model-based controller to real-world imperfections.

• Mechanism: The base controller (e.g., an Inverse Kinematics solver) provides a nominal action. The residual policy, observing state error, outputs a small delta.
• Benefit: Improves robustness to friction, payload variations, and calibration errors.
• Training: Often trained via reinforcement learning in a high-fidelity simulator before zero-shot real-world deployment.

A Diffusion Policy models action prediction as an iterative denoising process. It generates smooth, multi-step action sequences conditioned on visual observations.

• Strength: Excels at multi-modal action distributions (e.g., a peg can be inserted from multiple angles).
• Output: A T x A dimension tensor, where T is the action horizon and A is the action dimension.
• Application: Demonstrated for precise, contact-rich tasks like wiping a surface or assembling tight-fitting parts, where smoothness is critical.

In a multi-agent system, each robot runs a decentralized policy network. These policies take local observations but are trained to exhibit cooperative behavior.

• Observation: Each agent's view (camera, LiDAR) and communication from neighbors.
• Training: Uses Multi-Agent Reinforcement Learning (MARL) paradigms like Centralized Training with Decentralized Execution.
• Example: A fleet of warehouse robots learning collision-free navigation and coordinated item transport without a central planner.

A hierarchical policy is a multi-level control architecture that decomposes complex tasks into manageable sub-tasks. A high-level manager policy selects among temporally extended skill primitives or sub-policies, which in turn execute sequences of low-level actions. This abstraction improves sample efficiency, enables long-horizon planning, and facilitates skill reuse.

• Key Mechanism: Temporal abstraction via options or skills.
• Example: A high-level policy chooses 'open drawer', triggering a pre-trained low-level policy for precise gripper positioning and pulling.
• Benefit: Addresses the credit assignment problem over long time horizons.

A goal-conditioned policy is a control function, π(a | s, g), that generates actions based on both the current state (s) and a specified goal (g). This formulation enables a single, versatile policy to achieve a wide range of objectives without retraining.

• Key Mechanism: Conditions policy output on a goal representation (e.g., target image, state vector, language instruction).
• Training: Often uses self-supervised learning where goals are randomly sampled from achieved states in a replay buffer.
• Application: Essential for instruction-following robots and multi-task reinforcement learning.

Diffusion policy is a recent robot learning approach that formulates action prediction as a conditional denoising process. Instead of outputting actions directly, the network models a series of denoising steps that iteratively refine a sequence of noisy actions to match demonstrated behavior.

• Key Mechanism: Models the gradient of the data distribution (score function) and uses it to reverse a diffusion process.
• Advantages: Excels at modeling multi-modal action distributions (multiple good ways to perform a task) and generates smooth, realistic action sequences.
• Architecture: Often uses a Vision Transformer (ViT) or CNN to process observations and a U-Net for the denoising steps.

The Decision Transformer is a transformer-based architecture that models sequential decision-making as conditional sequence generation. It frames reinforcement learning as a supervised sequence modeling problem.

• Key Mechanism: Takes a trajectory of past states, actions, and desired return-to-go (a scalar reward target) as an input sequence. It autoregressively predicts future actions.
• Paradigm Shift: Replaces reward maximization with goal-conditioned imitation learning on offline datasets.
• Benefit: Leverages the powerful sequence modeling capabilities of transformers and simplifies training by avoiding dynamic programming.

A visuomotor control policy is a specific type of policy network that maps raw or processed visual observations (e.g., camera pixels) directly to low-level motor commands (e.g., joint torques or velocities). It tightly couples perception and action.

• Key Challenge: Must learn robust visual representations invariant to lighting, viewpoint, and object appearance.
• Architecture: Typically an end-to-end neural network combining a convolutional encoder for vision and a multi-layer perceptron or recurrent network for control.
• Use Case: The core of many imitation learning and reinforcement learning systems for robotic manipulation.

A value network is a neural network that estimates the expected cumulative future reward (the value) of being in a given state (V(s)) or of taking a specific action in a state (Q(s,a)). It is the primary companion to a policy network in actor-critic reinforcement learning algorithms.

• Role: Provides a training signal (critic) to the policy network (actor) by evaluating the quality of states and actions.
• Algorithm Example: In Proximal Policy Optimization (PPO), the value network loss is mean-squared error between predicted and actual returns.
• Contrast with Policy: The policy network decides what to do; the value network estimates how good the current situation is.