Policy Conditioning

Policy conditioning fundamentally alters the policy network's architecture by adding the randomized simulation parameters as an explicit input vector. This creates a context-aware controller where the mapping from state to action is explicitly conditioned on the current domain's configuration.

• Input Concatenation: The domain parameter vector (e.g., [mass=1.2, friction=0.7, lighting=0.4]) is concatenated with the standard state observation (e.g., joint angles, velocities).
• Adaptive Behavior: The network learns distinct control strategies for different parameter sets, effectively internalizing a manifold of possible dynamics models.
• Contrast with Standard DR: Unlike standard Domain Randomization, where the policy must be robust to all variations implicitly, a conditioned policy learns to explicitly identify and adapt to the current variation.

This technique shifts the learning objective from implicit robustness to explicit adaptation. A standard DR-trained policy must output a single action that works adequately across the entire randomization distribution. A conditioned policy, however, outputs an action specific to the provided parameters.

• Implicit Robustness (Standard DR): The policy's weights encode an average, compromise solution. Performance may be sub-optimal for any specific parameter setting but acceptable across the range.
• Explicit Adaptation (Policy Conditioning): The policy's forward pass computes a tailored solution. This can lead to higher performance within the trained distribution, as the network is not forced to average.
• Trade-off: Conditioning requires the policy to learn a more complex function but can yield more precise control when the context (domain parameters) is known or can be estimated.

A primary advantage of policy conditioning is its compatibility with online system identification. Since the policy is designed to accept domain parameters, it can be deployed with a parallel process that estimates those parameters from real-world data.

• Closed-Loop Adaptation: A real robot can run a lightweight system ID algorithm (e.g., Bayesian inference, recursive least squares) to estimate current dynamics parameters (like friction) from its sensor stream.
• Parameter Injection: These estimated parameters are then fed into the conditioned policy, allowing it to adjust its control strategy in real-time to match the identified physical conditions.
• Bridging the Reality Gap: This creates a powerful sim-to-real pipeline: train in randomized simulation with conditioning, then deploy with online parameter estimation to adapt to the unseen real world.

The effectiveness of policy conditioning depends heavily on the design and representation of the randomization input vector. Not all parameters are equally informative or learnable.

• Parameter Selection: Engineers must choose which simulation parameters to condition on. Key candidates are those with high real-world variability and significant impact on dynamics (e.g., payload mass, surface friction, motor gain).
• Normalization and Scaling: Input parameters must be normalized to a consistent scale (e.g., [0,1] or zero-mean, unit-variance) to ensure stable gradient flow during training.
• Discrete vs. Continuous: Conditioning can be applied to both continuous parameters (e.g., coefficient of restitution) and discrete categorical variables (e.g., terrain type represented as a one-hot vector).

Policy conditioning introduces distinct operational modes between training in simulation and deployment in the real world.

• Training Mode: During each training episode, the simulator samples a parameter vector θ from the randomization distribution. This vector is provided as input alongside the state. The policy learns the function π(a | s, θ).
• Inference Mode (Known Parameters): If the real-world parameters are known (e.g., a calibrated robot on a known surface), the corresponding vector is fed to the policy for optimal performance.
• Inference Mode (Estimated Parameters): As described, parameters can be estimated online from sensor data and used as input.
• Inference Mode (Default/Mean Parameters): If no estimation is available, feeding the mean or median of the training distribution can provide a robust fallback, often outperforming a standard DR policy.

Policy conditioning is conceptually aligned with few-shot meta-learning or context-based meta-learning. The randomized simulation parameters act as the "context" that defines a specific task within a distribution of related tasks.

• Task Distribution: Each set of randomized parameters (e.g., high friction, low mass) defines a unique instance of the control task.
• Learning to Adapt: The policy network learns a higher-order function that can quickly specialize its behavior based on the context, similar to how a meta-learner adapts to a new task from few examples.
• Rapid Online Fine-Tuning: This perspective suggests that a well-conditioned policy could be fine-tuned on the real robot with very few episodes, as it already has a strong prior over the space of possible dynamics.

The foundational technique where simulation parameters (e.g., physics, visuals, sensor noise) are randomly varied during training. This creates a diverse set of source domains, forcing the policy to learn generalizable strategies. Policy conditioning builds upon this by explicitly providing these parameters as input.

• Core Idea: Expose the model to many possible "worlds."
• Goal: Achieve zero-shot transfer to the real world.
• Example: Training a robot arm with randomized object mass, surface friction, and motor latency.

An algorithmic extension that dynamically expands the randomization range during training. Instead of using a fixed distribution, ADR automatically adjusts parameter bounds to focus on challenging scenarios where the policy fails, creating a curriculum of increasing difficulty.

• Mechanism: Uses policy failure as a signal to widen the randomization space.
• Advantage: Reduces the need for manual tuning of randomization bounds.
• Relation to Policy Conditioning: ADR can generate the parameter vectors that are fed into a conditioned policy network.

The process of estimating the true parameters of a physical system (the target domain) from observed data. This is often seen as a complementary or alternative approach to domain randomization.

• Contrast with Randomization: Aims to minimize the reality gap by making the simulation more accurate, rather than training a policy to be robust to inaccuracy.
• Hybrid Approach: System ID can be used to define the center or bounds of a randomization distribution, making it more physically plausible.
• Policy Conditioning Link: A conditioned policy could, in theory, receive identified real-world parameters as its input during deployment to specialize its behavior.

A broader class of policies that adapt their behavior based on context variables. Policy conditioning is a specific instantiation where the context is the domain's randomized parameters.

• General Principle: Policy π(a | s, c) where 'c' is the context.
• Other Contexts: Could be task specifications, user preferences, or high-level commands.
• Architecture: Often implemented using hypernetworks or feature-wise linear modulation (FiLM) layers to modulate the base policy based on context.

The ultimate objective of techniques like domain randomization and policy conditioning. A robust policy maintains high performance across a wide range of environmental variations and uncertainties, including those encountered in the real world (out-of-distribution robustness).

• Evaluation: Measured by performance across a randomized simulation ensemble or, ultimately, by sim2real success rate on physical hardware.
• Trade-off: There is often a performance trade-off between optimality in a single domain and robustness across many.
• Worst-Case Domain: Robust optimization often aims to improve performance under the most challenging conditions within the randomization range.

An alternative technique for domain generalization that uses an adversarial loss to learn features that are invariant to the domain. A domain classifier tries to distinguish which domain a feature comes from, while the feature extractor tries to fool it.

• Key Difference vs. Policy Conditioning: Seeks to remove domain-specific information from features, whereas policy conditioning explicitly provides it.
• Use Case: Often applied in vision-based tasks where domain-invariant visual features are desired.
• Combination Possible: Policy conditioning can be used in tandem with domain-adversarial training on other parts of the network.