Imitation Learning from Observations (IfO)

The defining constraint of Imitation Learning from Observations (IfO). The learning agent has access only to sequences of expert states (e.g., joint angles, object positions) from demonstrations, without the corresponding expert actions (e.g., motor torques, control signals). The algorithm must infer or recover the missing action information to learn a viable policy. This is a more realistic but challenging setting, as actions are often unobservable in third-party videos or biological motion capture.

• Primary Data: State sequences (s_0, s_1, ..., s_T).
• Missing Data: Action labels (a_0, a_1, ..., a_{T-1}).
• Implication: Precludes direct supervised learning (behavioral cloning) and necessitates more complex inference.

A primary technical strategy to address the missing action problem. The agent learns an inverse dynamics model—a function that predicts the action taken given a current state and a next state: a_t = f(s_t, s_{t+1}). This model is trained using data generated by the agent's own interactions with the environment.

• Process: 1) The agent explores the environment, collecting (s_t, a_t, s_{t+1}) tuples. 2) Train the inverse model on this data. 3) Apply the trained model to expert state transitions (s_t^expert, s_{t+1}^expert) to impute probable expert actions.
• Challenge: The model must generalize accurately from the agent's potentially suboptimal exploration data to the expert's state distribution.

The core optimization objective in many IfO algorithms. Instead of matching actions, the goal is to match the state visitation distribution of the learner's policy to that of the expert. The agent learns a policy whose resulting trajectories of states are indistinguishable from the expert's observed state sequences.

• Formal Goal: Minimize a divergence (e.g., Jensen-Shannon) between d_π(s) (learner's state distribution) and d_E(s) (expert's state distribution).
• Adversarial Approach: Often implemented using a GAN-like framework, where a discriminator is trained to distinguish between learner-generated state sequences and expert state sequences, while the policy (generator) is trained to fool it.
• Benefit: Bypasses the need to explicitly recover actions at training time.

A major practical application and advantage of the IfO paradigm. Because it requires only state observations, it can learn from demonstrations captured from a different viewpoint or embodiment, such as watching human videos. This is also known as Third-Person Imitation Learning.

• Key Challenge: The domain shift between the expert's viewpoint/embodiment and the agent's own sensors. The observed states s_t^expert (pixel coordinates in a video) are different from the agent's egocentric states s_t^agent (proprioceptive readings).
• Solution: Requires an additional representation learning or domain adaptation step to map third-person observations into the agent's own state space before distribution matching or inverse dynamics can be applied.

IfO shares a fundamental similarity with Inverse Reinforcement Learning (IRL): both infer latent objectives from observed behavior. In IRL, the latent object is a reward function. In IfO, the latent information is the action policy (or the dynamics necessary to compute it).

• Connection: Many IfO algorithms are derived from or are special cases of IRL frameworks (e.g., Maximum Entropy IRL) where the action is marginalized out or the reward is defined on states only.
• Distinction: IfO's explicit goal is a policy, not a reward function. However, learning a state-based reward via IRL and then using RL to recover a policy is a valid IfO approach, though often less efficient than direct policy learning.

A significant learning challenge inherent to the setting. Without action labels, it is harder for the algorithm to discern the causal relationship between states and the expert's decisions. The agent may learn spurious correlations or passive behaviors that coincide with the expert's state sequence but do not constitute the correct policy.

• Example: An expert robot arm moves to avoid an obstacle. The state sequence shows the arm moving left as the obstacle approaches. An IfO agent might learn to move left periodically, correlating with obstacle presence, rather than learning the causal rule "move away from nearby objects."
• Mitigation: Requires robust exploration during inverse model training or the use of temporal consistency constraints in the distribution matching objective to favor causal, actionable policies.

Behavioral Cloning is a supervised learning approach where a policy is trained to directly map observed states to the expert's actions, minimizing prediction error on a static dataset. It is the simplest form of imitation learning but suffers from compounding errors due to covariate shift when the learner deviates from the expert's state distribution. IfO can be seen as a more challenging variant where the action labels are missing, forcing the algorithm to infer them.

Inverse Reinforcement Learning is the problem of inferring a reward function from observed expert behavior, under the principle that the demonstrations are optimal with respect to some unknown reward. Unlike IfO, IRL typically assumes access to expert actions. IfO can be a stepping stone to IRL, where first the missing actions are inferred, and then the reward is recovered. Both fields grapple with reward ambiguity, where many rewards can explain the same behavior.

Third-Person Imitation Learning addresses the viewpoint mismatch problem, where demonstrations are captured from a perspective different from the agent's own (e.g., a robot learning by watching a human). Like IfO, it deals with partial observability of the expert's process. IfO focuses on missing action information, while third-person imitation focuses on a different visual or state representation. Solutions often involve learning viewpoint-invariant or domain-invariant feature spaces.

Adversarial Imitation Learning frames imitation as a distribution matching problem, using a discriminator network to distinguish between trajectories generated by the learner and the expert. Generative Adversarial Imitation Learning (GAIL) is a prominent example. IfO algorithms often adopt this adversarial framework but must operate without action pairs, matching only state transition distributions. The discriminator is trained to differentiate state sequences, not state-action pairs.

Inverse Optimal Control is the classical control theory counterpart to IRL, focused on inferring a cost function from optimal trajectories in deterministic, often model-based, settings. It is closely related to the foundations of IfO, as recovering a cost function from state-only observations requires solving a two-step problem: 1) estimating the unobserved control inputs, and 2) inferring the cost that makes those inputs optimal. IOC provides a rigorous mathematical basis for understanding the IfO problem.

Visual Imitation Learning involves training a policy from high-dimensional visual observations, such as videos. This is a common and practical setting for IfO, where the demonstrations are raw video pixels without action labels. The challenge combines representation learning—extracting relevant state features from pixels—with the core IfO problem of action inference. Successful approaches often use embodied vision-language models or self-supervised techniques to learn actionable representations from visual sequences.