Diffusion Policy

The policy does not predict actions directly. Instead, it is trained to iteratively denoise a sequence of random Gaussian noise into a coherent action trajectory, conditioned on the current observation (e.g., an image). This process starts from pure noise and applies the trained model over T steps to produce the final action sequence. The reverse diffusion process inherently smooths out predictions, leading to more stable and consistent outputs compared to direct regression.

To ensure smooth, physically plausible motions, Diffusion Policy predicts a temporal "chunk" of actions (e.g., 1-2 seconds of future actions) per inference, rather than a single time step. The policy is conditioned on a window of past observations. At execution, a receding horizon controller selects the first few actions from the predicted chunk, then re-plans. This chunk-based prediction explicitly models action dependencies, mitigating the compounding error common in step-by-step Behavior Cloning.

The diffusion process naturally captures multi-modal action distributions. Since the model learns to denoise from many possible noise samples to match the data distribution, a single observation can yield diverse, valid action trajectories. This is critical for tasks where multiple strategies are viable (e.g., grasping an object from different angles). Unlike deterministic policies or those using a fixed Mixture of Experts, sampling different noise vectors provides a principled way to explore the space of expert behaviors.

The denoising process is conditioned on the current perceptual context. This is typically achieved by feeding fused observation features (e.g., from a vision backbone processing camera images) into the diffusion model's cross-attention or feature-wise linear modulation (FiLM) layers. The model learns to invert the forward diffusion process only for trajectories consistent with the observed scene, grounding the action prediction in the current robot state and environment.

The policy outputs a full action sequence across multiple degrees of freedom (DoF). For a 6-DoF robot arm, this might be a chunk of K timesteps, each with 6 values for joint positions or end-effector pose. The diffusion model's U-Net architecture processes this as a 2D tensor (Time × Action-Dim), allowing it to learn correlations across both time and different joint axes, producing coordinated movements.

Training uses the standard denoising score matching objective. Given an expert action sequence a_0, noise ε is added for t diffusion steps to produce a_t. The model (a U-Net) is trained to predict the added noise ε given a_t, the diffusion step t, and the observation o. The loss is simple mean-squared error: L = || ε - ε_θ(a_t, t, o) ||^2. No adversarial training or complex reward functions are required, leading to stable optimization.

Diffusion Policy is exceptionally effective for contact-rich manipulation tasks where actions are inherently multi-modal. For example, a robot learning to insert a peg into a hole can predict multiple plausible force-application strategies. The iterative denoising process allows the policy to converge on a smooth, physically plausible trajectory that accounts for contact dynamics, outperforming deterministic policies that often produce jittery or infeasible motions in such scenarios.

This architecture is ideal for learning from human demonstrations (e.g., via teleoperation or motion capture). Human demonstrations contain natural variability; a single task can be performed in several valid ways. A Diffusion Policy, by modeling a distribution over actions, can capture this multi-modality instead of averaging behaviors into a single, potentially ineffective mean action. This leads to more robust and human-like skill reproduction.

By predicting action sequences (not just single steps), Diffusion Policy enables temporally coherent planning. For a task like "unstack the red block and place it on the blue one," the policy conditions on the entire visual history and language instruction to denoise a coherent multi-step plan. This chaining of actions reduces the compounding error common in single-step behavior cloning, as the model is trained to output trajectories that remain consistent over time.

When trained in simulation with domain randomization, Diffusion Policies exhibit strong sim-to-real generalization. The stochastic denoising process can be interpreted as exploring the action space to find robust strategies that work across randomized physics parameters, lighting, and textures. This makes the policy less sensitive to the reality gap, as it learns a distribution over actions that are valid under many environmental conditions.

Diffusion Policy can be adapted for Learning from Observations (LfO), where only state sequences (e.g., video of a human) are available, not actions. The model is trained to predict the next state instead of an action, and an inverse dynamics model can be used to infer the required control signals. The diffusion process helps manage the ambiguity inherent in inferring actions from states alone.

As a visuomotor policy, Diffusion Policy acts as the low-level action decoder within a Vision-Language-Action (VLA) model. A high-level VLA model (e.g., a large language or vision-language model) processes instructions and images to produce a high-level plan or goal embedding. This embedding conditions the Diffusion Policy, which then generates the precise, smooth motor commands needed for physical execution, bridging abstract reasoning to concrete actuation.

A supervised learning approach where a policy is trained to directly map observed states to actions by minimizing error against an expert dataset. It is the simplest form of imitation learning but suffers from compounding error when the agent encounters states not in the training distribution.

• Key Mechanism: Treats imitation as a standard regression/classification problem.
• Limitation: Assumes independent and identically distributed (i.i.d.) data, which breaks during sequential rollouts.

An adversarial imitation learning framework that frames imitation as a distribution matching problem. A generator (the policy) learns to produce trajectories that a discriminator cannot distinguish from expert demonstrations.

• Key Mechanism: Uses a minimax game to align the learner's state-action visitation distribution with the expert's.
• Contrast with Diffusion Policy: GAIL is adversarial and reward-based; Diffusion Policy is a conditional generative model that denoises action trajectories directly.

An iterative, online algorithm designed to mitigate the compounding error problem in Behavior Cloning. It collects corrective actions from an expert on states visited by the learner's current policy, aggregating them into an expanding training dataset.

• Key Mechanism: Iterates between policy rollouts and expert relabeling to cover the learner's state distribution.
• Requirement: Needs ongoing access to an expert or oracle during training, which can be costly.

The paradigm of learning a policy from a fixed, pre-collected dataset of demonstrations without any further environment interaction or expert queries during training. This is the standard setting for applying Diffusion Policy.

• Key Constraint: The policy must learn effectively from a static dataset, avoiding distributional shift.
• Advantage: Safe and data-efficient, as it doesn't require risky online exploration.

A trajectory representation that models a distribution over movements, capturing variability and correlations across degrees of freedom and time. It provides a compact, probabilistic encoding of demonstrated skills.

• Key Mechanism: Represents a trajectory via a weighted sum of basis functions, with a distribution over the weights.
• Contrast with Diffusion Policy: ProMP is a parameterized probabilistic model; Diffusion Policy is a denoising generative model that can represent more complex, multi-modal distributions.

A family of algorithms that use adversarial training to match distributions between the learner and expert. The core idea is to learn a policy such that its generated trajectories are indistinguishable from expert ones to a trained discriminator.

• Core Members: Includes GAIL, AIRL (Adversarial Inverse Reinforcement Learning), and their variants.
• Theoretical Basis: Often connected to minimizing the Jensen-Shannon divergence or Wasserstein distance between distributions.