Learning from Observations (LfO)

The foundational constraint of LfO is the demonstration dataset, which contains only sequences of environmental states (e.g., robot joint angles, object positions) visited by the expert. Crucially, the expert's actions (e.g., motor torques, control signals) are not recorded or provided. This is common when demonstrations come from videos of humans or other agents where direct action telemetry is unavailable.

To learn a policy, the agent must infer which action could have caused the observed state transition. This is achieved by learning an inverse dynamics model (IDM), a function that predicts the action taken given a current state and a next state: a_t = f(s_t, s_{t+1}). The IDM is typically a neural network trained on data collected by the agent itself through exploration, creating a self-supervised learning loop to understand its own embodiment.

LfO systems often separate learning into two distinct phases:

• Phase 1: Inverse Dynamics Pre-training. The agent explores its environment, collecting tuples of (s_t, a_t, s_{t+1}) to train a general-purpose inverse dynamics model.
• Phase 2: Policy Learning from States. The pre-trained IDM is used to label the expert's state-only demonstrations with inferred actions. These pseudo-labeled (s_t, â_t) pairs are then used to train the policy via standard supervised learning (e.g., Behavior Cloning).

A major technical hurdle is action ambiguity or the one-to-many mapping problem. A single observed state transition (s_t -> s_{t+1}) can often be achieved by multiple different actions. For example, moving a cup from point A to B can be done with a fast or slow grip, or with different wrist orientations. The inverse dynamics model must learn a disambiguating representation, often by leveraging temporal context or learning a distribution over plausible actions.

LfO sits within a spectrum of imitation learning techniques:

• vs. Behavior Cloning (BC): BC requires state-action pairs. LfO is BC's harder cousin, dealing with incomplete data.
• vs. Inverse Reinforcement Learning (IRL): Both learn from states, but IRL aims to recover a reward function, while LfO aims to recover a policy directly.
• vs. Adversarial Imitation (e.g., GAIL): GAIL typically uses state-action pairs. LfO variants like Observation-Conditioned GAIL adapt the discriminator to work on state-only distributions.

LfO is motivated by practical scenarios where action data is inaccessible or ill-defined:

• Learning from Human Video: The largest source of demonstrations (e.g., YouTube, instructional videos) provides visual state sequences but not human muscle commands.
• Cross-Embodiment Learning: Demonstrations from a physically different agent (e.g., a human) cannot provide directly executable low-level motor commands for a robot.
• Legacy or Proprietary Systems: When the expert is a black-box controller or legacy system where action signals cannot be intercepted.

LfO enables robots to learn from watching humans, bypassing the need for teleoperation or kinesthetic teaching. This is critical for tasks where direct action recording is infeasible.

• Key Challenge: The embodiment gap between human and robot morphologies.
• Solution: Learn an inverse dynamics model to infer plausible robot actions from observed human state transitions.
• Example: A robot arm learning to assemble furniture by watching YouTube videos of humans, using video frames to estimate object poses (states) and learning which motor commands achieve those pose changes.

LfO can leverage vast archives of state-only telemetry from operational systems (e.g., autonomous vehicles, industrial machines) where the original control commands were not logged or are proprietary.

• Process: Treat historical state sequences as unsupervised demonstrations.
• Technical Approach: Use self-supervised learning to train a world model that predicts future states, then derive a policy via planning or latent action inference.
• Business Value: Turns passive observational data into an actionable control policy without needing to instrument legacy systems for action recording.

A policy trained via LfO on one robotic platform (the demonstrator) can be transferred to a different platform (the learner) with different actuators and dynamics, provided both share a similar state representation.

• Core Mechanism: The policy learns the intent of state transitions rather than platform-specific motor commands.
• Requirement: A shared state space (e.g., end-effector pose, object positions) is essential.
• Example: A policy learned from a precise industrial robot's motions (states) can be adapted to guide a cheaper, less precise robot by re-learning the inverse dynamics specific to the new hardware.

This is the core technical component that makes LfO possible. It is a learned function that predicts the action taken between two consecutive states.

• Function: a_t = f(s_t, s_{t+1})
• Training: Requires a separate, often small, dataset of state-action pairs collected by the learner agent through random exploration or auxiliary tasks.
• At Inference: The learned policy outputs a target next state. The inverse dynamics model then calculates the action needed to achieve that transition on the specific learner hardware.

Modern LfO approaches often use generative models to predict plausible future state trajectories directly, avoiding explicit inverse dynamics.

• Diffusion Models: A Diffusion Policy can be trained to denoise a sequence of future states conditioned on the current state and goal.
• Variational Autoencoders (VAEs): Learn a latent space where state sequences can be decoded, and a policy operates in this latent space.
• Advantage: These models naturally handle multi-modality (multiple valid next states) and generate smoother, more coherent long-horizon plans.

LfO is a powerful tool for sim-to-real transfer. An expert policy can be trained in simulation (generating state trajectories), and a physical robot can learn from these "observations" using its own inverse dynamics.

• Workflow:
  1. Train an expert policy in simulation (with access to actions).
  2. Record its state-only trajectories.
  3. Use these as demonstrations for LfO on the real robot.
• Benefit: Decouples the simulation's possibly inaccurate dynamics model from the real robot's control, as the real robot learns its own action-generation mechanism.

An Inverse Dynamics Model is a learned function that predicts the action taken to transition between two consecutive states. It is the core technical component enabling Learning from Observations (LfO).

• Function: Given states s_t and s_{t+1}, it predicts a_t.
• Role in LfO: It is trained on trajectories generated by the agent's own exploration or a random policy, then used to label the state-only expert demonstrations with inferred actions, converting an LfO problem into a standard Behavior Cloning problem.
• Challenge: The model must generalize accurately from the agent's potentially limited exploration data to the expert's state distribution.

State-Only Demonstrations are the fundamental input data for LfO. They consist of sequences of environment states (s_0, s_1, ..., s_T) recorded during expert task execution, with the corresponding expert actions (a_0, a_1, ..., a_T) intentionally omitted or unavailable.

• Sources: Human videos, motion capture systems, or telemetry from a different embodiment.
• Key Advantage: Vastly expands the potential sources of demonstration data, as recording actions often requires instrumented hardware or direct access to an expert's control interface.
• Core Problem: Creates a correspondence problem between states and the missing actions that caused the transitions.

Inverse Reinforcement Learning (IRL) is a related but distinct paradigm for learning from expert behavior. While LfO learns a policy directly from states, IRL infers the underlying reward function that explains the expert's behavior, assuming optimality.

• Contrast with LfO: IRL typically requires access to expert state-action pairs, not just states. It outputs a reward function, which then guides reinforcement learning. LfO outputs a policy directly.
• Connection: Some advanced LfO methods blend the approaches, using adversarial or contrastive techniques to match state visitation distributions without explicitly recovering actions, similar to the distribution-matching objective of IRL methods like GAIL.

The Embodiment Gap is the mismatch in physical form, dynamics, and action spaces between the agent learning from demonstrations and the expert that generated them. This is a primary challenge LfO aims to address.

• Manifestation in LfO: Even with perfect state observations, a human's joint angles (state) cannot be directly mapped to a robot's different actuator commands (action).
• LfO's Advantage: By not relying on expert actions, LfO methods can, in principle, learn a policy that achieves the same state transitions using the agent's own action space, bridging the embodiment gap through the learned inverse dynamics model.
• Remaining Challenge: The gap can still exist at the state level if the expert's state representation is incompatible (e.g., human skeleton vs. robot proprioception).

A Forward Dynamics Model predicts the next state s_{t+1} given the current state s_t and action a_t. It is often learned in tandem with an inverse dynamics model in LfO frameworks.

• Synergistic Role: Learning a forward model provides a useful supervisory signal for representation learning. By predicting the outcome of actions, the agent learns a more structured and actionable state representation.
• Use in Planning: A trained forward model allows the agent to perform model-based planning or imagination rollouts to refine its policy or recover from errors, complementing the imitation objective.
• Joint Training: Many LfO architectures use a composite loss, training inverse and forward models concurrently to learn a coherent, dynamics-aware latent space.

Observation-Only Imitation is a direct synonym for Learning from Observations (LfO). It emphasizes the defining constraint: the learning agent has access only to the observable outcomes of expert behavior, not the expert's control signals.

• Practical Implications: This paradigm is essential for scaling robot learning to real-world settings where action data is scarce, such as learning from:
  • Internet videos of humans performing tasks.
  • Historical sensor logs from factories or vehicles.
  • Third-party robotic systems with proprietary or inaccessible controllers.
• Research Frontier: Advances in this area focus on overcoming the temporal credit assignment problem—determining which latent actions were responsible for observed state changes over long sequences.