Goal-Conditioned Policy

The defining feature is that the goal specification is provided as an explicit input to the policy network, alongside the current observation (e.g., an image). This goal can be represented in multiple ways:

• State Goal: A desired configuration (e.g., joint angles, object positions).
• Image Goal: A target image of the scene.
• Language Goal: A natural language instruction (e.g., "pick up the blue block").
• Embedding Goal: A latent vector from a higher-level planner. The policy learns a mapping: π(action | observation, goal).

Instead of learning a separate policy for each task, a single goal-conditioned policy is trained to handle a distribution of goals. This promotes generalization to unseen but related goals within the training distribution. For example, a policy trained to reach various points on a table can often reach a new, interpolated point without explicit training. This is a form of zero-shot generalization enabled by the continuous goal space.

These policies are trained using algorithms that explicitly incorporate the goal. Common methods include:

• Goal-Conditioned Reinforcement Learning (GCRL): Algorithms like Hindsight Experience Replay (HER) relabel failed trajectories with achieved goals as targets, drastically improving sample efficiency in sparse-reward settings.
• Goal-Conditioned Imitation Learning: Training via supervised learning on demonstrations where each state-action pair is paired with the final goal of the trajectory.
• Multi-Task Learning: Training on a curated set of tasks/goals, often with a shared network backbone and task-specific heads or embeddings.

Goal-conditioned policies are often used as low-level skills or primitive actions within a hierarchical policy. A high-level planner or manager selects a sub-goal (e.g., "grasp tool") at a coarser time scale, and the goal-conditioned policy executes the detailed motor commands to achieve it. This decouples long-horizon reasoning from precise, reactive control.

A major challenge is learning a useful goal representation. For complex goals like images or language, an encoder network is used to project the goal into a latent space that aligns with the observation space. This often involves contrastive learning or metric learning to ensure that similar goals are close in the latent space, and that the distance to the goal can be used as a learning signal.

A key technique, especially in Hindsight Experience Replay, is goal relabeling. When a trajectory fails to achieve its original goal, it can be treated as a successful demonstration for the goal it did achieve. This transforms every trajectory, successful or not, into a valid training example for multiple goals, providing a powerful data augmentation mechanism that is critical for learning in real-world robotics where success is initially rare.

In warehouse automation, a single goal-conditioned policy can control a robotic arm to pick and place diverse objects. The goal is specified as a target 3D coordinate or a semantic label (e.g., 'pick the red box').

• The policy uses visual input to locate the target object among clutter.
• It generalizes across different object shapes, sizes, and positions.
• This eliminates the need for separate, task-specific programs for each SKU, enabling flexible automation.

Autonomous mobile robots (AMRs) in hospitals or factories use goal-conditioned policies for navigation. The goal is a room number or GPS coordinate.

• The policy processes LiDAR and camera data to perceive obstacles and hallways.
• It outputs low-level velocity commands to navigate to dynamically specified destinations.
• This allows a single deployed policy to handle thousands of possible delivery endpoints without retraining.

This is a core use case for Vision-Language-Action (VLA) models. A human provides a natural language instruction like 'put the apple in the bowl.'

• The language command is encoded into a goal embedding.
• The policy, conditioned on this embedding, visually grounds the objects ('apple', 'bowl') and generates the precise motor sequence for the manipulation task.
• This enables intuitive human-robot interaction for complex, multi-step chores.

Goal-conditioned policies are extensively trained in physics simulators using reinforcement learning. Randomizing goals during training (e.g., different block stacking configurations) builds robustness.

• The learned policy can then be transferred to a physical robot via domain randomization techniques.
• The robot can execute the same variety of tasks in the real world, bridging the sim2real gap for multi-task operation.

In complex tasks, a high-level policy sets sub-goals (e.g., 'open drawer', 'grasp tool'). A low-level goal-conditioned policy then executes these sub-goals.

• The low-level policy is a universal 'skill' that achieves any provided sub-goal within its capability.
• This decomposition makes learning long-horizon tasks (like 'make coffee') more tractable and composable.

For assistive devices, a goal can be personalized user intent. A robotic arm could be conditioned on goals like 'bring the water bottle' or 'turn the page.'

• The policy learns from user demonstrations via imitation learning.
• It can then adapt to the user's specific environment and preferences, providing reliable assistance for activities of daily living.

A policy structured into multiple levels of abstraction, where a high-level policy selects temporally extended sub-goals or skills, which are then executed by a dedicated low-level policy. This decomposition is a common architectural pattern for implementing goal-conditioned behavior.

• High-Level Planner: Operates at a coarse time scale, outputting abstract goals (e.g., 'pick up the block', 'navigate to the kitchen').
• Low-Level Controller: A goal-conditioned policy that receives the sub-goal and executes the precise motor commands to achieve it over a shorter horizon.
• Benefits: Improves sample efficiency, enables skill reuse, and simplifies learning complex, long-horizon tasks by breaking them into manageable parts.

The paradigm of inferring an underlying reward function from demonstrations of expert behavior, rather than learning a policy from an engineer-specified reward. IRL is a foundational technique for learning goal-conditioned policies when the goal is implicit in the demonstration.

• Process: Given a set of trajectories, IRL algorithms estimate the reward function that makes the expert behavior appear optimal.
• Connection to GCPs: The inferred reward function can then be used to train a goal-conditioned policy via reinforcement learning. Alternatively, the demonstrations can be relabeled with inferred goals for supervised or offline RL training.
• Key Challenge: IRL is an ill-posed problem—many reward functions can explain the same behavior.

A core conceptual and algorithmic innovation that directly enables deep goal-conditioned reinforcement learning. A UVFA is a single value function (e.g., Q-function) that is conditioned on both the state and an explicit goal.

• Function: (Q(s, a, g)) estimates the expected return of taking action (a) in state (s) when pursuing goal (g).
• Generalization: By approximating value for many (state, goal) pairs, the agent can generalize its knowledge to new goals not seen during training.
• Policy Derivation: A goal-conditioned policy can be derived by acting greedily with respect to the UVFA: (\pi(a|s, g) = \arg\max_a Q(s, a, g)).

A critical data augmentation technique for training goal-conditioned policies in sparse-reward environments. HER repurposes failed experiences as successful ones by relabeling them with alternative goals.

• Mechanism: In a trajectory where the agent did not achieve its original goal (g), HER stores additional transitions in the replay buffer where the goal is replaced with a state actually achieved later in the episode ((g' = s_T)).
• Effect: The transition ((s_t, a_t, r_t, s_{t+1}, g')) now has a positive reward, providing a learning signal. This teaches the policy that actions can be useful for achieving different goals.
• Impact: Dramatically improves sample efficiency and enables learning in environments where the original goal is rarely reached.

A representation learning framework that decouples the dynamics of the environment from the reward function, facilitating fast adaptation to new goals. It is particularly powerful for goal-conditioned policy learning when goals are defined by reward functions.

• Representation: An agent learns features (\psi(s, a, s')) that capture the expected future occupancy of successor states, independent of any specific reward.
• Goal Adaptation: Any new goal, defined by a reward function (r_g(s, a, s')), can be expressed as a linear combination of these features. The value function for the new goal is computed instantly via a dot product: (Q_g(s, a) = \psi(s, a)^T w_g), where (w_g) are the reward weights.
• Benefit for GCPs: Enables a single policy to rapidly evaluate progress towards a vast set of new goals without retraining, by simply changing the weight vector (w_g).

The broader training paradigm where a single agent learns to perform multiple distinct tasks. Goal-conditioned reinforcement learning is a specific, generalizable instantiation of multi-task RL.

• Standard Multi-Task RL: The agent is trained on a fixed set of (N) tasks, each with its own reward function (r_i). The policy or value function is often conditioned on a task identifier (i).
• Goal-Conditioned as Generalization: By conditioning on a goal (g) (which can be a target state, image, or description) instead of a task ID, the agent can, in principle, generalize to an infinite or unseen set of tasks at test time.
• Key Distinction: Multi-task learning aims for high performance on the training task distribution. Goal-conditioned learning aims for the emergent ability to achieve any specified goal from a goal space.