Trajectory Matching

The L2 norm, or Mean Squared Error (MSE), is the most common pointwise distance metric for trajectory matching. It calculates the average squared difference between corresponding states or actions in the learner's and expert's trajectories.

• Mechanism: For a trajectory of length T, MSE = (1/T) Σ ||s_learner(t) - s_expert(t)||².
• Use Case: Dominates Behavior Cloning where the policy is trained via supervised regression.
• Limitation: Assumes temporal alignment and is sensitive to outliers. It fails to account for temporal warping or global shape similarity.

Dynamic Time Warping (DTW) is an algorithm that measures similarity between two temporal sequences which may vary in speed. It finds an optimal non-linear alignment (warping path) between them by minimizing the cumulative distance.

• Mechanism: Computes a cost matrix and finds the path with minimal cumulative cost, allowing one timestep in one sequence to match multiple in the other.
• Use Case: Essential for matching human and robot demonstrations with different velocities or pauses.
• Advantage: Invariant to local temporal shifts, making it robust for matching trajectories with different timings.

The Frechet distance (or 'dog-walking' distance) is a measure of similarity between curves that accounts for the location and ordering of points. It is defined as the minimum leash length required for a point traversing the learner's trajectory to stay connected to a point traversing the expert's trajectory.

• Mechanism: Considers the geometry of the curves and the sequence of points. The discrete Frechet distance is commonly used for polygonal curves.
• Use Case: Evaluates the geometric fidelity of navigation or manipulation paths, where the global shape is more critical than pointwise error.
• Property: More geometrically meaningful than DTW for some applications, as it is a metric satisfying the triangle inequality.

When demonstrations are treated as distributions (e.g., using Probabilistic Movement Primitives), trajectory matching minimizes the divergence between probability distributions. The Kullback-Leibler (KL) Divergence is a fundamental measure for this.

• Mechanism: KL(P_expert || P_learner) measures the information loss when using the learner's distribution to approximate the expert's. Minimizing it matches the moments (mean, covariance) of the trajectories.
• Use Case: Critical in Bayesian or variational imitation learning frameworks that capture demonstration uncertainty.
• Benefit: Enables learning of robust, generalizable policies that capture the variability inherent in expert data.

Generative Adversarial Imitation Learning (GAIL) reframes trajectory matching as a distribution matching problem. Instead of minimizing a predefined distance, it uses a discriminator network to distinguish between expert and learner state-action pairs, and a generator (policy) learns to fool it.

• Mechanism: The policy's objective is to minimize the discriminator's ability to classify its trajectories as 'generated,' effectively minimizing the Jensen-Shannon divergence between distributions.
• Use Case: Overcomes limitations of compounding error in Behavior Cloning by learning from on-policy interactions.
• Outcome: The policy learns to match the expert's state-action visitation distribution, not just individual trajectories.

Dynamic Movement Primitives are a trajectory representation and matching method. A DMP encodes a demonstrated trajectory as a stable dynamical system defined by a set of differential equations with a forcing function.

• Mechanism: The forcing function is learned via locally weighted regression to reproduce the demonstration. Matching is achieved by minimizing the error between the learned and target forcing functions.
• Use Case: Provides a compact, modular representation for skills that can be easily scaled, shifted, and combined.
• Key Feature: The dynamical system formulation guarantees convergence to a goal, providing inherent stability during execution.

Behavior Cloning is the most direct form of imitation learning, treating it as a supervised learning problem. A policy network is trained to map observed states directly to expert actions by minimizing a loss (e.g., Mean Squared Error) on a static dataset of demonstrations.

• Primary Challenge: Suffers from compounding error due to covariate shift; small errors cause the agent to visit states not in the training distribution, leading to failure.
• Use Case: Effective for short-horizon tasks or for initializing policies for further fine-tuning with online methods.

Inverse Reinforcement Learning addresses a fundamental limitation of pure trajectory matching: it seeks to infer the underlying reward function the expert is optimizing, rather than just copying actions.

• Core Idea: Assumes the expert is (near-)optimal. The algorithm finds a reward function that makes the expert's behavior appear optimal, then uses RL to train a policy with that reward.
• Advantage: Can generalize better to new situations than BC, as it understands the intent behind the demonstrations.
• Example: Inferring that a self-driving car's reward values smooth turns and maintaining a safe distance, not just the exact steering angles recorded.

GAIL is a prominent Adversarial Imitation Learning framework that frames imitation as a distribution matching problem, bypassing explicit reward inference.

• Mechanism: It jointly trains a policy (generator) and a discriminator. The policy learns to produce trajectories that are indistinguishable from expert trajectories to the discriminator, which learns to tell them apart.
• Outcome: The policy's state-action visitation distribution converges to the expert's, effectively learning the reward implicitly through the discriminator's signals.
• Benefit: Often more stable and sample-efficient than IRL for complex, high-dimensional tasks.

DAgger is an online algorithm designed to combat the compounding error problem in Behavior Cloning by iteratively collecting corrective data.

• Process:
  1. Train an initial policy on the expert dataset.
  2. Roll out the current policy in the environment (or simulation).
  3. Query the expert (or an oracle) for the correct action in the states actually visited by the policy.
  4. Aggregate this new corrective data with the old dataset and retrain.
• Result: The training distribution is gradually shifted to match the learner's induced state distribution, dramatically reducing covariate shift.

A Dynamic Movement Primitive is a robust policy representation commonly used with trajectory matching, especially in robotics. It encodes a trajectory as a stable dynamical system.

• Key Properties:
  • Stability: Guaranteed convergence to a specified goal.
  • Temporal Scalability: The trajectory can be sped up or slowed down without changing its shape.
  • Modulation: The trajectory can be easily adjusted to new start or goal points.
• Usage in Imitation: A DMP is first learned from a single or multiple demonstrated trajectories (matching the shape). It can then be executed as a policy, providing smooth, generalizable motor commands.

These are the two central, interrelated failure modes that imitation learning algorithms, including trajectory matching, must overcome.

• Compounding Error: In sequential decision-making, a small error at time step t leads the agent to a state s_{t+1} slightly different from the demonstration. The policy, trained on s_{t+1} from the demo, is now in unfamiliar territory, leading to a larger error, which compounds over the horizon.
• Covariate Shift: This is the statistical manifestation of compounding error. The distribution of states encountered by the learned policy p_π(s) diverges from the distribution of states in the expert dataset p_data(s). Since the policy was trained on p_data(s), its performance degrades on p_π(s). Algorithms like DAgger are explicitly designed to correct this shift.