Unity ML-Agents

The core abstraction is a Unity Scene configured as a training environment. Key components include:

• Agents: GameObjects that observe, decide, and act.
• Academy: Singleton that orchestrates the training loop and environment reset logic.
• Decision Requester: Component that automatically requests decisions from an agent's policy at a defined frequency.
• Behavior Parameters: Links an agent to a specific trained model or heuristic policy, defining its observation space and action space.

Training is orchestrated from Python using the mlagents package, which communicates with the Unity runtime via a gRPC-based communicator. This separation allows:

• Leveraging standard ML libraries like PyTorch for algorithm implementation.
• Distributed training across multiple environment instances for faster, more stable learning.
• The Unity executable to act as a rendering and physics backend, streaming observations and receiving actions.

The toolkit provides implementations of state-of-the-art model-free reinforcement learning algorithms:

• Proximal Policy Optimization (PPO): A robust, on-policy algorithm for stable policy gradient updates.
• Soft Actor-Critic (SAC): An off-policy algorithm that maximizes entropy, excellent for continuous control tasks.
• Behavioral Cloning: A supervised imitation learning method to bootstrap policies from demonstration data.
• GAIL (Generative Adversarial Imitation Learning): A more advanced imitation learning technique that learns a reward function from demonstrations.

Agents perceive their world through multiple observation types, which can be fused:

• Vector Observations: Numeric data (e.g., position, velocity, inventory) passed directly to the policy network.
• Visual Observations (Camera Sensors): One or more camera feeds processed by Convolutional Neural Networks (CNNs).
• Raycast Observations: Simulated lidar-like distance measurements to detect nearby objects.
• Grid Sensors: A structured spatial grid capturing categorical data (e.g., object type) in the agent's vicinity.

Agents output actions defined by their action space:

• Continuous Actions: A vector of floating-point numbers (e.g., torque, steering angle).
• Discrete Actions: A set of mutually exclusive choices (e.g., move left/right/jump).
• Hybrid Actions: A combination of both continuous and discrete branches for complex behaviors. Actions are applied via Actuator Components, which translate neural network outputs into forces, rotations, or other game logic.

Critical techniques for sim-to-real transfer and robust policy learning:

• Curriculum Learning: Training starts with easy tasks; difficulty increases automatically as the agent masters objectives.
• Environment Parameter Randomization: Physical properties (mass, friction), visual appearances (textures, lighting), and object positions are varied during training. This forces the policy to learn generalizable strategies, bridging the reality gap for deployment on physical systems.

Beyond pure RL, ML-Agents supports imitation learning paradigms where agents learn by observing demonstrations, which is crucial for tasks where designing a reward function is difficult.

• Behavior Cloning: Train a supervised policy directly from recorded state-action pairs of a human controlling the agent in-engine.
• Inverse Reinforcement Learning (IRL): Infer the underlying reward function that explains expert behavior, then optimize a policy for that reward.
• Interactive Training: Use the Heuristic Brain to manually control an agent during training, providing real-time demonstrations for the learning algorithm.

The toolkit excels at simulating environments with multiple interacting agents, enabling research into cooperation, competition, and communication.

• Decentralized Training: Train individual agents with independent or shared policies.
• Complex Emergent Behavior: Study how simple local rules learned by individual agents can lead to complex global group behaviors like flocking, teamwork, or market dynamics.
• Adversarial Training: Co-train agents in competitive scenarios (e.g., hide-and-seek, soccer) to drive the emergence of advanced strategies and robust policies.

ML-Agents is used to build digital twin prototypes for autonomous vehicles, warehouse robots, and smart factories, allowing for risk-free testing of high-level decision-making algorithms.

• Scenario Testing: Simulate rare or dangerous edge cases (e.g., sensor failure, adverse weather) to validate system robustness.
• Hardware-in-the-Loop (HIL): Interface the simulation with real control hardware for validation.
• High-Fidelity Visualization: Leverage Unity's rendering for realistic visual output, which is critical for testing perception systems and for stakeholder demonstrations.

The critical process of deploying a policy trained in a Unity simulation onto a physical robot. This involves overcoming the reality gap—discrepancies between simulation and real-world physics/sensors. ML-Agents supports techniques like Domain Randomization, where visual textures, lighting, and physical parameters (e.g., mass, friction) are varied during training to force the policy to generalize to a wider distribution of conditions, improving robustness for real-world deployment.

A training paradigm where an agent learns by mimicking expert demonstrations, often used when designing a suitable reward function is difficult. ML-Agents implements:

• Behavior Cloning: A supervised learning approach that directly maps observed states to expert actions.
• More advanced inverse reinforcement learning methods. This is particularly valuable for bootstrapping complex behaviors or providing a safe initial policy for further RL fine-tuning.

The core software that simulates Newtonian dynamics within Unity, enabling realistic agent-environment interaction. ML-Agents leverages Unity's built-in engines (NVIDIA PhysX, others) or can interface with high-precision alternatives like MuJoCo. The engine calculates:

• Rigid body dynamics and collisions.
• Contact forces, friction, and joints. The fidelity and performance of the physics engine are paramount for training physically plausible visuomotor control policies and enabling accurate sim-to-real transfer.

Fundamental RL constructs that define an agent's interface with the Unity environment.

• Observation Space: The sensory input the agent receives. In ML-Agents, this can be vector (e.g., joint angles, raycast distances), visual (camera RGB/D images), or a combination (multi-modal).
• Action Space: The set of possible outputs from the agent's policy. ML-Agents supports continuous spaces (e.g., motor torques) and discrete spaces (e.g., move left/jump), or hybrid branching actions for complex decision-making.