Probabilistic Movement Primitive (ProMP)

Unlike deterministic primitives, a ProMP models a probability distribution over possible trajectories for a movement skill. This is typically represented as a Gaussian distribution over the trajectory parameters. This allows the model to:

• Capture the natural variability observed in multiple demonstrations of the same task.
• Represent correlations between different joints (degrees of freedom) and across time.
• Generate smooth, stochastic movements by sampling from the distribution.

Trajectories are represented as a weighted sum of temporal basis functions (e.g., normalized Radial Basis Functions). The movement is defined as: y_t = Φ_t * w + ε where y_t is the observed trajectory (position/velocity), Φ_t is the basis matrix, w are the weights, and ε is noise. The weights w are treated as random variables with a distribution p(w; θ). This formulation provides a compact representation and enables easy temporal scaling and shifting of the movement by modulating the phase variable.

A key strength of ProMPs is the ability to condition the trajectory distribution on observed via-points or partial trajectories using Bayes' theorem. This enables:

• Online adaptation to new goals or via-points.
• Reactive behavior where the robot can adjust its movement mid-execution upon receiving new sensor information (e.g., an obstacle appears).
• Blending of skills by conditioning on the final state of one primitive to initialize the next. The conditional distribution p(w | y_t*) is analytically tractable because the model is Gaussian.

ProMPs explicitly model and allow control over movement variance and the coupling between joints. This is crucial for robotics:

• High variance can be commanded for exploratory movements or less critical phases.
• Low variance can be enforced for precise, repeatable actions like insertion.
• The full covariance matrix captures how deviations in one joint correlate with deviations in another and across time, preserving the coordinated structure of the demonstrated skill.

ProMPs address several limitations of the earlier Dynamic Movement Primitive (DMP) framework:

• DMPs are deterministic; ProMPs are probabilistic, capturing variability.
• DMPs use nonlinear attractor dynamics for stability; ProMPs use probabilistic conditioning for adaptation.
• Coupling between degrees of freedom is more naturally represented in ProMPs via the covariance matrix, whereas DMPs typically treat them independently.
• ProMPs provide a unified framework for learning, conditioning, and blending within probability theory.

In imitation learning, ProMPs are used to encode multiple demonstrations of a task. The learning process involves:

1. Collecting N demonstrations of a skill.
2. Learning the parameters θ (mean and covariance of the weight distribution p(w)) that best explain the observed data, typically via maximum likelihood estimation.
3. The resulting ProMP represents the essential features of the skill, filtering out demonstration noise while retaining intentional variability. This model can then be executed, conditioned, and blended for robust skill execution.

A Dynamic Movement Primitive (DMP) is a deterministic, differential equation-based framework for representing motor skills as stable nonlinear attractor systems. Unlike ProMPs, DMPs model a single, canonical trajectory. Key features include:

• Stability Guarantees: Inherent convergence to a goal via a spring-damper system.
• Temporal Scaling: The trajectory's duration can be stretched or compressed without reshaping.
• Goal and Trajectory Modulation: The final goal and the shape of the movement path can be altered independently.
• Robotics Use Case: DMPs are widely used for point-to-point reaching and simple periodic motions, where a single, reproducible trajectory is sufficient.

Behavior Cloning (BC) is a supervised learning method where a policy (e.g., a neural network) is trained to directly map observed states to actions by minimizing the error between its predictions and the actions taken by an expert in a dataset of demonstrations. ProMPs are often used as the policy representation within a BC framework. Key distinctions:

• ProMP in BC: The ProMP's weight distribution is learned from demonstrations, and sampling from this distribution generates stochastic trajectories.
• Compounding Error: A core limitation of naive BC; small errors cause the agent to enter states not seen during training, leading to failure. ProMP's probabilistic nature can model the variability needed to be more robust to such errors.

Generative Adversarial Imitation Learning (GAIL) is an adversarial framework where a generator (the policy) learns to produce behavior that a discriminator cannot distinguish from expert demonstrations. It directly matches state-action visitation distributions. Relation to ProMPs:

• ProMP as Generator: A ProMP can serve as the stochastic policy (generator) in GAIL, providing a structured, low-dimensional representation from which to sample trajectories.
• Distribution Matching: Both ProMP and GAIL aim to model the distribution of expert behavior, not just a single trajectory. GAIL does this implicitly via the discriminator, while ProMP does it explicitly via a probability distribution over trajectory parameters.

Trajectory Optimization is a planning technique that computes a sequence of states and actions minimizing a cost function while satisfying system dynamics and constraints (e.g., torque limits, obstacles). It is complementary to ProMPs:

• Optimization for Refinement: A trajectory optimized for a specific scenario (e.g., a new goal position) can serve as a single demonstration for training or adapting a ProMP.
• ProMP for Initialization/Warm-Start: The mean trajectory from a ProMP can provide an excellent initial guess for a trajectory optimizer, significantly speeding up convergence.
• Combined Approach: A common pipeline uses trajectory optimization offline to generate multiple demonstrations for varying conditions, which are then used to learn a generalizable ProMP.

In imitation learning, a Mixture of Experts (MoE) is a modular policy architecture where multiple sub-policies ("experts") are combined via a gating network. This relates to ProMPs in modeling complex, multi-modal behaviors:

• ProMP as an Expert: A single ProMP can represent a primitive skill (e.g., "reach for cup").
• Hierarchical Skill Libraries: Multiple ProMPs can form a library of movement primitives. An MoE gating network can then select or blend these ProMPs based on the task context, enabling the composition of complex skills from simpler, reusable components.
• Multi-Modal Distributions: For tasks with several distinct successful strategies, a mixture of ProMPs can explicitly model the multi-modality better than a single Gaussian ProMP.

Covariate Shift is a fundamental problem in imitation learning where the state distribution encountered by the learner's policy during execution diverges from the state distribution present in the expert's demonstration dataset. This mismatch causes performance degradation. ProMPs address this by:

• Modeling Variability: By learning a distribution over trajectories, ProMPs inherently capture the expert's variance, including how to behave in slightly off-distribution states.
• Conditional Generation: A trained ProMP can be conditioned on the current state (via probabilistic inference) to generate an appropriate continuation of the movement, even if the start state differs from the typical demonstration start.