Goal-Conditioned Policy

The defining feature is the explicit inclusion of a goal representation as an input to the policy function, alongside the current state observation. This goal can be specified in various forms:

• A target end-effector pose in Cartesian space.
• A desired image of a scene configuration.
• A natural language instruction (e.g., "place the blue block on the red one").
• A target state vector or a set of achieved conditions. The policy π(a | s, g) learns to map the current state s and goal g to an action a that progresses the agent toward g.

A properly trained goal-conditioned policy exhibits strong generalization to goals not seen during training, provided they are within the learned goal distribution. This is a key advantage over training separate policies for each task. The policy learns a relational understanding of state and goal space, enabling it to infer the necessary action sequence for novel goal specifications. This capability is fundamental for versatile robotic systems that must operate in open-world environments.

Instead of maintaining a library of task-specific controllers, a single neural network architecture (e.g., a transformer or multilayer perceptron) is trained to handle all goals. This simplifies system design and deployment. Common architectures include:

• Goal-Conditioned Value Networks used in reinforcement learning (e.g., Hindsight Experience Replay).
• Goal-Conditioned Behavioral Cloning networks trained on demonstration data labeled with goals.
• Decision Transformers that take a sequence of states, actions, and a goal/return-to-go as input.

Training often relies on a dense reward function r(s, a, g) that provides incremental feedback on progress toward the goal, rather than a sparse success/failure signal. A common approach is to use a distance metric in the goal space. For example:

• Reward = -||φ(s') - g||, where φ(s') is the achieved goal features. This dense shaping guides the policy to learn smooth trajectories toward any specified target g.

Goal-conditioned policies form the foundation of hierarchical reinforcement learning (HRL). In HRL, a high-level manager policy sets sub-goals in a learned or abstract space. A low-level, goal-conditioned policy (or skill primitive) then executes actions to achieve each sub-goal. This decouples long-horizon planning from low-level control, making complex task decomposition feasible.

A major engineering challenge is designing an appropriate goal representation that is both informative for the policy and generalizable. Poor representations lead to failed generalization. Common approaches include:

• Using a VLA model's latent space to embed language or image goals.
• State-based goals: A vector of desired sensor readings or object positions.
• Image-based goals: A target RGB-D image for visual servoing. The choice dictates what the policy can be commanded to do and how it perceives task completion.

A Universal Value Function Approximator (UVFA) is a core theoretical framework that enables goal-conditioned learning. Instead of learning a separate value function for each goal, a UVFA is a single neural network that approximates the value function for any state-goal pair (s, g). This generalization is fundamental to training efficient goal-conditioned policies, as it allows the agent to estimate how good any state is for achieving any specified goal, facilitating robust planning and credit assignment across a continuum of objectives.

Hindsight Experience Replay (HER) is a critical reinforcement learning algorithm for training goal-conditioned policies in sparse reward environments. When an agent fails to achieve its intended goal g, HER relabels the experience in the replay buffer as if the goal had been a different one that was actually achieved—often the final state of the episode. This creates a powerful supervised learning signal from what would otherwise be a failure, dramatically improving sample efficiency. For example, a robot failing to grasp a red block might have moved the blue block; HER treats this as a successful episode for the goal 'move blue block'.

Goal-Conditioned Supervised Learning (GCSL) is an alternative to RL that frames policy learning as a supervised regression problem. Given a dataset of trajectories (s_t, a_t, s_{t+1}), GCSL relabels each state-action pair with a goal that the action helped achieve (e.g., a future state in the same trajectory). The policy is then trained via behavioral cloning to predict the action a_t given the current state s_t and the relabeled goal g. This approach bypasses the need for reward engineering and value function learning, often leading to more stable and efficient training for complex manipulation tasks.

Goal representation refers to the form in which an objective is specified to a goal-conditioned policy. The choice of representation is crucial for generalization and learning efficiency.

• State-Based Goals: The goal g is a desired configuration of the environment state (e.g., joint angles, object positions). This is common in robotics.
• Image-Based Goals: The goal is a target image of the desired scene. The policy must learn to map current visual observations to actions that minimize the difference between the current and goal image.
• Language-Based Goals: The goal is a natural language instruction (e.g., 'put the mug on the coaster'). This requires the policy to integrate a vision-language model to ground the instruction in visual perception and action space.

Hierarchical Reinforcement Learning (HRL) decomposes long-horizon tasks using a multi-level policy structure, which is closely related to goal-conditioned control. In HRL, a high-level manager policy sets sub-goals for a low-level worker policy over extended time horizons. The low-level policy is inherently goal-conditioned, as it learns to execute the sub-goal specified by the manager. This abstraction allows the system to tackle complex tasks by breaking them into sequences of simpler, goal-directed skills, improving exploration and temporal credit assignment.

Multi-Task Reinforcement Learning trains a single agent to perform multiple distinct tasks, each with its own reward function. A goal-conditioned policy is a specific instantiation of multi-task learning where the task is explicitly defined by the goal g. The key distinction is that in pure multi-task learning, the task might be identified by a discrete label (e.g., Task ID 3: 'open drawer'), whereas goal-conditioning typically uses a continuous goal representation. The policy's architecture must be designed to prevent catastrophic interference, where learning a new task degrades performance on previously learned ones.