Continuous Action Space MCTS

This technique integrates local optimization routines into the simulation (rollout) phase of MCTS to handle continuous parameters.

• Instead of using a random or simple heuristic policy for the entire rollout, the algorithm can call a fast, local optimizer (e.g., a gradient-based method, cross-entropy method, or a learned policy network) from the expanded node to generate a sequence of actions.
• The rollout then follows this locally optimized trajectory to a terminal state or a depth limit. The resulting reward is backpropagated, evaluating the potential of the region of the action space defined by the selected node.
• This is particularly effective in model-based settings or when a differentiable dynamics model is available, allowing for efficient gradient computation during the rollout.

Value Function Approximation replaces the Monte Carlo sampling of rollouts with a learned model that estimates the expected return from a given state-action pair.

• A value network or Q-network is trained (often concurrently with search) to approximate Q(s, a) for continuous actions a. During MCTS selection and backpropagation, this network provides value estimates, drastically reducing the need for long, expensive random rollouts.
• In the expansion phase, a policy network can suggest promising continuous actions to add as child nodes, acting as an informed prior. This is the core of algorithms like AlphaZero, adapted for continuous control.
• This hybrid approach combines the planning strength of MCTS with the generalization and efficiency of neural networks, scaling to high-dimensional spaces like robotic manipulation.

Hierarchical Action Decomposition breaks down a complex, high-dimensional continuous action into a sequence of simpler decisions in a multi-level MCTS tree.

• A high-level MCTS planner might choose abstract, discrete skills or sub-goals (e.g., "grasp object," "move to location").
• Each of these abstract actions is then parameterized by a lower-level MCTS tree or a dedicated controller that plans the continuous parameters (e.g., joint angles, velocities) to achieve the sub-goal.
• This transforms a single search over a vast continuous space into a series of searches over smaller, more manageable spaces. It mirrors Hierarchical Task Network (HTN) planning and is key for Embodied Intelligence Systems where actions have temporal extent.

Continuous Action Space MCTS competes with and complements other planning and reinforcement learning methods for continuous control.

• Deep Deterministic Policy Gradient (DDPG) / Soft Actor-Critic (SAC): These model-free RL algorithms learn a direct policy and Q-function. MCTS is a planning algorithm that does not require explicit training but uses more online computation per decision.
• Model Predictive Control (MPC): MPC uses an explicit dynamics model to optimize a short-horizon action sequence via online optimization (e.g., shooting methods). MCTS with optimization rollouts is a sampling-based variant of MPC.
• Cross-Entropy Method (CEM): A zeroth-order optimization technique that samples action sequences. MCTS with progressive widening provides a more structured, tree-based exploration strategy that can better handle long-horizon, sparse-reward problems.
• The choice depends on the need for online planning speed, availability of a model, and horizon length.

A straightforward alternative to progressive widening where the continuous action space is quantized into a finite set of representative actions before applying standard MCTS.

• Fixed Discretization: A static grid or set of bins (e.g., 10 possible thrust levels for a rocket).
• Adaptive Discretization: The resolution of the discretization adapts based on the estimated value landscape, focusing more bins on promising regions.
• Limitation: Can suffer from the curse of dimensionality and may miss optimal actions that fall between the discrete bins.

Many continuous control problems solved with MCTS are framed within model-based RL, where the agent learns or is given a dynamics model of the environment.

• MCTS uses this model as a simulator for the rollout phase.
• The learned model can be a neural network (as in MuZero) predicting next state and reward.
• This combination allows for sample-efficient planning by imagining trajectories without costly real-world interaction.
• Continuous Action Space MCTS is often the planning module inside a larger model-based RL agent.

MuZero is a seminal algorithm that demonstrates MCTS in learned continuous action spaces. It extends AlphaZero to environments with unknown dynamics.

• It learns a latent dynamics model that predicts rewards, policy, and value in a hidden state space.
• The action space can be continuous; the neural network's policy head outputs parameters (e.g., mean of a distribution) for action sampling during tree search.
• Progressive widening is applied in this latent space, using the network to propose new action candidates.

Continuous Action Space MCTS belongs to the broader family of heuristic search algorithms designed for large state-action spaces.

• Differentiates from classic methods like A* or Dijkstra's, which require explicit graphs and admissible heuristics.
• Monte Carlo approach makes it suitable for problems where a good heuristic is unknown or the state space is stochastic.
• Other related algorithms include Cross-Entropy Method (CEM) for continuous optimization and Rapidly-exploring Random Trees (RRT) for robotic path planning, which share the theme of guided random sampling.