AlphaZero

AlphaZero learns entirely from self-play, starting with random moves. It treats the game as a Markov Decision Process, where the agent (the current player) interacts with the environment (the game board) by selecting actions (moves). Through millions of self-play games, it uses the REINFORCE algorithm to update its neural network parameters, maximizing the expected cumulative reward (winning the game). This creates a closed learning loop where the agent continuously improves by playing against progressively stronger versions of itself, without any prior human knowledge or labeled datasets.

AlphaZero uses a heuristic search algorithm to evaluate positions and select moves during both training and play. The MCTS builds a search tree incrementally through four phases:

• Selection: Traverse the tree from the root using the PUCT (Polynomial Upper Confidence Trees) formula, balancing explored nodes with high value (exploitation) and less-visited nodes (exploration).
• Expansion: Add a new child node to the tree when reaching a leaf node.
• Simulation (Rollout): Perform a fast rollout to a terminal game state using the neural network's policy head to sample moves, not random play.
• Backpropagation: Update the visit count and value estimate of all nodes along the traversed path with the game's outcome (+1 for win, -1 for loss, 0 for draw). This process generates improved policy targets for training, as MCTS explores more deeply than the raw network output.

A single, deep convolutional neural network with a dual-headed architecture serves as AlphaZero's core evaluator and guide.

• Input: A stack of 8-17 planar feature maps representing the current board state and recent history.
• Body: A residual tower of convolutional blocks (using batch normalization and ReLU activations) that extracts spatial features.
• Policy Head (p): Outputs a probability distribution over all legal moves, guiding the MCTS search.
• Value Head (v): Outputs a scalar estimate of the expected game outcome from the current player's perspective (win/loss/draw). The network is trained to minimize a combined loss function: (z - v)^2 - π^T log p + c ||θ||^2, where z is the actual game outcome, π is the improved search-based policy, and c is an L2 regularization constant. This unified model replaces the separate policy and value networks used in its predecessor, AlphaGo Zero.

AlphaZero's architecture is game-agnostic. The same algorithm, network topology, and hyperparameters (with minor tuning) mastered chess, shogi, and Go. Only the game rules are provided as input to define:

• The legal move generator.
• The game termination condition.
• The outcome scalar (win/loss/draw). The neural network's input planes are configured to represent game-specific state information (e.g., piece types and repetition counters for chess). This demonstrates a general-purpose reinforcement learning and planning framework applicable to any deterministic, perfect-information, two-player zero-sum game with a discrete action space.

Training was accelerated using a massive, distributed computing infrastructure. The system consisted of thousands of TPUs (Tensor Processing Units) for neural network inference and hundreds of GPUs for experience generation.

• Self-Play Actors: Many parallel actors played games, using the latest network checkpoint to generate training data (state, MCTS-improved policy π, outcome z).
• Training Learner: A single learner process continuously sampled batches of experience from a replay buffer, computed gradients, and updated the neural network weights.
• Checkpointing: Updated network parameters were periodically pushed to the self-play actors. This asynchronous pipeline enabled the generation of hundreds of millions of self-play games and efficient, stable training over several days.

Despite its ultimate superhuman performance, AlphaZero is sample-inefficient by modern deep RL standards but computationally efficient in terms of environmental complexity.

• Self-Play Games: It required ~44 million games of self-play over 9 hours (700,000 steps) to surpass AlphaGo Zero in Go, and ~1 million games to beat Stockfish in chess.
• No Domain Knowledge: It achieved this without handcrafted features, opening books, or endgame tables, learning complex concepts like king safety and pawn structure from scratch.
• Compute Scale: The training utilized ~5,000 first-generation TPUs and ~64 GPUs. However, the final evaluated model required only 4 TPUs for MCTS, demonstrating that the immense computational cost is front-loaded in training, not inference.

Monte Carlo Tree Search is the core planning algorithm used by AlphaZero. It is a heuristic search algorithm for optimal decision-making in sequential problems. MCTS builds a search tree incrementally by performing four key steps in a loop:

• Selection: Traverse the tree from the root using a selection policy (like UCT) to pick a child node.
• Expansion: Add one or more child nodes to the tree, expanding the frontier.
• Simulation (Rollout): Play out a simulated game from the new node(s) to a terminal state using a fast, default policy.
• Backpropagation: Update the statistics (visit count, value estimate) of all nodes along the traversed path with the simulation's result. AlphaZero's innovation was to replace the random rollout policy with a deep neural network's value and policy outputs, making simulations vastly more informative.

Upper Confidence Bound for Trees is the canonical selection formula used within MCTS to balance exploration and exploitation. For a given node, it selects the child j that maximizes: UCT(j) = Q(j) + c * sqrt( ln(N(parent)) / N(j) ) Where:

• Q(j) is the average reward (exploitation term).
• N(j) is the visit count for child j.
• N(parent) is the visit count for the parent node.
• c is an exploration constant. This formula ensures that promising nodes (high Q) are exploited, while less-visited nodes (low N) are explored. AlphaZero uses a variant of UCT where the prior probability from its neural network's policy head guides the exploration, heavily biasing the search toward moves the network considers strong.

Self-play is the training paradigm where an AI agent learns by competing against versions of itself, without any human data or intervention. AlphaZero's training loop consists of:

1. Iterative Improvement: The current best neural network plays thousands of games against a slightly older version of itself.
2. Data Generation: Each game produces a sequence of board positions, each labeled with the eventual game outcome (win/loss/draw) and the search-derived policy (visit counts from MCTS).
3. Network Training: The neural network is trained via supervised learning to predict both the game outcome (value) and the search probabilities (policy) for each position.
4. Iteration: The newly trained network becomes the next opponent. This creates a positive feedback loop, where the agent's policy and value estimates improve, which in turn improves the quality of the MCTS search, generating better training data.

AlphaZero uses a deep Residual Neural Network as its core architecture to process the game board. The ResNet is critical for its ability to train very deep networks effectively. Key features include:

• Residual Blocks: These blocks use skip connections that allow gradients to flow directly through the network, mitigating the vanishing gradient problem and enabling the training of networks with 40+ layers.
• Dual-Headed Output: The network has two output heads:
  • A policy head that outputs a probability distribution over all possible moves.
  • A value head that outputs a scalar estimating the expected game outcome from the current position (win probability).
• Input Representation: The network takes a stack of several recent board positions as input, providing a sense of game history. This architecture allows AlphaZero to evaluate positions with both strategic understanding (value) and tactical precision (policy).

AlphaZero is a quintessential example of a model-based reinforcement learning system. Unlike model-free RL (e.g., Q-learning), which learns a policy or value function directly from experience, model-based RL agents learn an internal model of the environment—in this case, the rules of the game—and use it for planning.

• Learned Model: AlphaZero's neural network implicitly encodes a dynamic model. The policy head predicts plausible actions, and the value head predicts outcomes, together forming a predictive model of state transitions and rewards.
• Planning with the Model: MCTS uses this learned model to perform lookahead search, simulating thousands of possible future game trajectories to evaluate the current position. This combination allows for extremely sample-efficient learning, as the agent can reason about consequences without experiencing every possible state directly.

AlphaZero is the direct successor to AlphaGo and AlphaGo Zero, representing the evolution of DeepMind's game-playing AI.

• AlphaGo (2016): Combined MCTS with supervised learning on human expert games and policy/value networks. It defeated world champion Lee Sedol.
• AlphaGo Zero (2017): Removed the dependency on human data, learning solely through self-play. It used a single neural network and defeated the original AlphaGo 100-0.
• AlphaZero (2017): Generalized the AlphaGo Zero algorithm to master multiple games—Chess, Shogi, and Go—using the same network architecture and hyperparameters (except input representation). It demonstrated the algorithm's domain-agnostic nature, achieving superhuman performance in all three games within 24 hours of training, starting from random play.