Dirichlet Noise

Dirichlet noise is sampled from the Dirichlet distribution, a multivariate generalization of the Beta distribution. It is defined by a concentration parameter vector (α). In the context of MCTS, a symmetric Dirichlet distribution is typically used, where α is a scalar, often set between 0.03 and 0.3 for games like Go. This distribution ensures that the sampled noise vector sums to 1, making it a natural choice for perturbing a probability distribution over actions.

• Parameter α: Controls the concentration of the distribution. A smaller α (e.g., 0.03) results in a sparse, peaky noise vector, favoring a few actions strongly. A larger α results in a more uniform noise vector.
• Noise Injection: The sampled Dirichlet noise vector is mixed with the neural network's prior policy output (P) at the root node: P' = (1 - ε) * P + ε * η, where η ~ Dir(α) and ε is a small mixing parameter (e.g., 0.25).

The core function of Dirichlet noise is to encourage exploration in the early moves of self-play games. Without it, the MCTS guided by a neural network's policy can rapidly converge to a narrow set of high-probability moves, leading to mode collapse where the agent only explores a limited subset of the state space.

• Counteracts Early Convergence: By adding stochasticity to the root node's priors, the search is forced to consider a wider variety of initial actions, even those deemed less likely by the current policy network.
• Discovering Novel Strategies: This forced exploration is crucial for discovering new, potentially superior strategies that the current policy model undervalues, driving the iterative improvement cycle in self-play reinforcement learning.

Dirichlet noise was a critical innovation in the AlphaZero algorithm. It is applied only at the root node of the MCTS during the self-play data generation phase.

• Root-Node Specific: Noise is not added during inference (competitive play) or to child nodes during search. This ensures exploratory data generation while retaining deterministic, exploitative play for evaluation.
• Combination with PUCT: The noise-perturbed prior probabilities (P') are used in the Polynomial Upper Confidence bound for Trees (PUCT) formula during the selection phase: U(s, a) = c_puct * P'(s, a) * (√N(s) / (1 + N(s, a))). This directly biases exploration towards the noisy prior.
• Empirical Values: In AlphaZero's Go play, typical parameters were α = 0.03 (for the 19x19 board) and ε = 0.25.

Dirichlet noise is uniquely suited for this task compared to other common noise types:

• vs. Gaussian Noise: Gaussian noise can produce negative values or cause the probability vector to not sum to 1, requiring renormalization. Dirichlet noise inherently generates a valid probability simplex.
• vs. Epsilon-Greedy: Epsilon-greedy makes a completely random choice with probability ε. Dirichlet noise provides a structured, yet stochastic, bias that still respects the relative strengths suggested by the policy network, just with added variance.
• vs. Temperature Scaling: Temperature scaling adjusts the entropy of the final action distribution after the search is complete. Dirichlet noise affects the search guidance itself by altering the priors at the start of the tree traversal.

The effectiveness of Dirichlet noise is sensitive to its two main hyperparameters: the concentration parameter (α) and the mixing coefficient (ε).

• Concentration Parameter (α): Must be scaled appropriately for the action space size. For a game with n legal moves at the root, a common heuristic is to use α = 1/n or a small constant. Too high an α makes the noise too uniform, diluting the policy's guidance. Too low makes it too sparse, causing excessive focus on a few random actions.
• Mixing Coefficient (ε): Balances the trust between the learned policy and the exploration noise. A value around 0.1 to 0.3 is typical. It often anneals over time in training schedules.
• Interaction with Visit Count: The influence of the noisy prior diminishes as a root node action is visited more, because the UCT formula's exploration term shrinks relative to the empirically derived value estimate.

Dirichlet noise interacts with several other MCTS and reinforcement learning mechanisms:

• Exploration-Exploitation Tradeoff: It is a direct, tunable mechanism for controlling this tradeoff specifically at the root of the search tree.
• Progressive Widening: In continuous or vast action spaces, Dirichlet noise at the root complements techniques like progressive widening, which gradually expands the number of child nodes considered.
• Self-Play Training Loop: The noise is a key component ensuring diversity in the training data generated by self-play, preventing the agent from becoming stuck in a local optimum by playing against a slightly varied version of itself.
• Policy Improvement Theorem: The noise facilitates exploration necessary for the policy iteration cycle to converge to an optimal policy, as it ensures all actions have a non-zero probability of being evaluated.

The fundamental dilemma in sequential decision-making of whether to sample new, uncertain actions (exploration) or to favor actions known to yield high rewards (exploitation). In MCTS, this is mathematically managed by selection policies like Upper Confidence Bound for Trees (UCT). Dirichlet noise directly injects stochasticity to bias this tradeoff towards exploration, especially in the early, uncertain phases of self-play training.

A hybrid architecture that integrates deep neural networks with classical MCTS. It typically uses two networks: a policy network to suggest promising actions during tree expansion and a value network to evaluate states, replacing random rollouts. Dirichlet noise is applied to the prior probabilities output by the policy network at the root node, ensuring the neural guidance does not prematurely collapse exploration.

A seminal self-play reinforcement learning system that combines a deep residual neural network with Monte Carlo Tree Search. It achieves superhuman performance in games like chess, shogi, and Go. Dirichlet noise was a critical innovation in AlphaZero, added to the root node's prior probabilities during training to ensure sufficient exploration of opening moves, preventing the agent from overfitting to a narrow set of early-game strategies.

In neural MCTS, the prior probability (P(s, a)) is an estimate, provided by a trained policy network, of the likelihood that action a is optimal from state s. This prior guides the initial search. Dirichlet noise perturbs these priors at the root node: P'(s, a) = (1 - ε) * P(s, a) + ε * η, where η is sampled from a Dirichlet distribution and ε is a small mixing parameter. This ensures the search tests a wider range of initial actions.

A technique used in MCTS for environments with large or continuous action spaces. Instead of expanding all possible child nodes at once, the algorithm gradually increases the number of considered actions for a parent node as its visit count grows. Dirichlet noise addresses a related but distinct problem: even with a small, discrete action set (like Go's 19x19 board), a strong policy network can assign near-zero prior to valid but novel moves, which progressive widening alone cannot fix.

A reinforcement learning paradigm where an agent improves by playing games against iterations of itself. This is the core training loop of AlphaZero and MuZero. Dirichlet noise is exclusively applied during these self-play games to generate diverse, exploratory training data. It is typically not used during competitive evaluation or real-world deployment, where the agent should exploit its learned knowledge deterministically.