Dirichlet Noise

Dirichlet noise was a critical innovation in the AlphaZero algorithm. It is applied only at the root node of the MCTS during the self-play data generation phase.

• Root-Node Specific: Noise is not added during inference (competitive play) or to child nodes during search. This ensures exploratory data generation while retaining deterministic, exploitative play for evaluation.
• Combination with PUCT: The noise-perturbed prior probabilities (P') are used in the Polynomial Upper Confidence bound for Trees (PUCT) formula during the selection phase: U(s, a) = c_puct * P'(s, a) * (√N(s) / (1 + N(s, a))). This directly biases exploration towards the noisy prior.
• Empirical Values: In AlphaZero's Go play, typical parameters were α = 0.03 (for the 19x19 board) and ε = 0.25.

Dirichlet noise is uniquely suited for this task compared to other common noise types:

• vs. Gaussian Noise: Gaussian noise can produce negative values or cause the probability vector to not sum to 1, requiring renormalization. Dirichlet noise inherently generates a valid probability simplex.
• vs. Epsilon-Greedy: Epsilon-greedy makes a completely random choice with probability ε. Dirichlet noise provides a structured, yet stochastic, bias that still respects the relative strengths suggested by the policy network, just with added variance.
• vs. Temperature Scaling: Temperature scaling adjusts the entropy of the final action distribution after the search is complete. Dirichlet noise affects the search guidance itself by altering the priors at the start of the tree traversal.

The effectiveness of Dirichlet noise is sensitive to its two main hyperparameters: the concentration parameter (α) and the mixing coefficient (ε).

• Concentration Parameter (α): Must be scaled appropriately for the action space size. For a game with n legal moves at the root, a common heuristic is to use α = 1/n or a small constant. Too high an α makes the noise too uniform, diluting the policy's guidance. Too low makes it too sparse, causing excessive focus on a few random actions.
• Mixing Coefficient (ε): Balances the trust between the learned policy and the exploration noise. A value around 0.1 to 0.3 is typical. It often anneals over time in training schedules.
• Interaction with Visit Count: The influence of the noisy prior diminishes as a root node action is visited more, because the UCT formula's exploration term shrinks relative to the empirically derived value estimate.

Dirichlet noise interacts with several other MCTS and reinforcement learning mechanisms:

• Exploration-Exploitation Tradeoff: It is a direct, tunable mechanism for controlling this tradeoff specifically at the root of the search tree.
• Progressive Widening: In continuous or vast action spaces, Dirichlet noise at the root complements techniques like progressive widening, which gradually expands the number of child nodes considered.
• Self-Play Training Loop: The noise is a key component ensuring diversity in the training data generated by self-play, preventing the agent from becoming stuck in a local optimum by playing against a slightly varied version of itself.
• Policy Improvement Theorem: The noise facilitates exploration necessary for the policy iteration cycle to converge to an optimal policy, as it ensures all actions have a non-zero probability of being evaluated.

A seminal self-play reinforcement learning system that combines a deep residual neural network with Monte Carlo Tree Search. It achieves superhuman performance in games like chess, shogi, and Go. Dirichlet noise was a critical innovation in AlphaZero, added to the root node's prior probabilities during training to ensure sufficient exploration of opening moves, preventing the agent from overfitting to a narrow set of early-game strategies.

In neural MCTS, the prior probability (P(s, a)) is an estimate, provided by a trained policy network, of the likelihood that action a is optimal from state s. This prior guides the initial search. Dirichlet noise perturbs these priors at the root node: P'(s, a) = (1 - ε) * P(s, a) + ε * η, where η is sampled from a Dirichlet distribution and ε is a small mixing parameter. This ensures the search tests a wider range of initial actions.

A technique used in MCTS for environments with large or continuous action spaces. Instead of expanding all possible child nodes at once, the algorithm gradually increases the number of considered actions for a parent node as its visit count grows. Dirichlet noise addresses a related but distinct problem: even with a small, discrete action set (like Go's 19x19 board), a strong policy network can assign near-zero prior to valid but novel moves, which progressive widening alone cannot fix.

A reinforcement learning paradigm where an agent improves by playing games against iterations of itself. This is the core training loop of AlphaZero and MuZero. Dirichlet noise is exclusively applied during these self-play games to generate diverse, exploratory training data. It is typically not used during competitive evaluation or real-world deployment, where the agent should exploit its learned knowledge deterministically.