Reinforcement Learning for Retrosynthesis

The retrosynthetic problem is formalized as an MDP where:

• State (s): The current target molecule represented as a molecular graph or SMILES string.
• Action (a): Selecting a valid disconnection and the resulting reactant set.
• Transition: The deterministic decomposition of the target into precursors.
• Reward (r): A scalar signal evaluating the quality of the chosen disconnection. This formulation allows the agent to learn a policy π(a|s) that maps molecules to optimal disconnections.

Unlike template-based classification, RL agents must solve the temporal credit assignment problem. A disconnection that appears poor locally may lead to an elegant convergent synthesis later.

• The agent maximizes cumulative discounted return, not immediate step-wise scores.
• This naturally penalizes dead-end pathways that lead to unsynthesizable intermediates.
• The discount factor γ controls the trade-off between short routes and high-confidence steps.

Agents are typically trained using policy gradient methods like REINFORCE or Proximal Policy Optimization (PPO):

• The policy network outputs a probability distribution over valid disconnections.
• The loss function weights log-probabilities by the advantage of the chosen action.
• This directly optimizes the expected quality of the synthetic route without requiring a value function for every intermediate.
• Entropy regularization is often added to prevent premature convergence to suboptimal reaction classes.

The reward function is the critical design element that encodes synthetic chemistry heuristics:

• Convergence bonus: Rewards disconnections that split the molecule into fragments of roughly equal size.
• Building block termination: High positive reward for reaching commercially available starting materials.
• Step penalty: A small negative reward per step to encourage shorter linear sequences.
• In-Scope Filtering: Negative rewards for reactions outside the model's validated chemical space.

RL agents balance exploitation of known high-yielding disconnections with exploration of novel bond-breaking strategies:

• ε-greedy strategies force the agent to occasionally sample low-probability reactions.
• Boltzmann exploration samples actions proportionally to their policy probabilities.
• This is crucial for discovering non-obvious umpolung or C-H activation disconnections that template-based systems would miss.
• The exploration schedule typically decays over training as the policy converges.

RL-trained policies serve as the rollout policy or prior for Monte Carlo Tree Search during inference:

• The learned policy biases the search towards high-value branches, dramatically reducing the effective branching factor.
• The value network estimates the synthesizability of intermediate nodes without full expansion.
• This hybrid approach mirrors AlphaGo-style architectures, combining learned intuition with systematic lookahead.
• The result is a significant reduction in the number of simulated rollouts required to find a viable route.

A heuristic search algorithm that forms the backbone of many RL-based retrosynthetic planners. MCTS builds an asymmetric search tree by iteratively running four steps:

• Selection: Traverse the tree from the root to a leaf using a tree policy (e.g., UCB1)
• Expansion: Add a new child node by applying a valid disconnection
• Simulation: Roll out a random policy to a terminal state to estimate value
• Backpropagation: Update node statistics with the simulation result MCTS balances exploration of unknown disconnections with exploitation of known high-value pathways, making it ideal for the vast chemical search space.

A search strategy specifically suited to retrosynthesis where nodes have logical constraints:

• AND nodes: Represent a reaction where all child reactants must be successfully synthesized for the reaction to succeed
• OR nodes: Represent a molecule where only one viable synthetic route needs to be found This structure naturally mirrors convergent synthesis, where multiple fragments are made independently and coupled. RL agents learn to navigate this tree by assigning Q-values to OR nodes and cost estimates to AND nodes.

A hierarchical data structure representing the recursive deconstruction of a target molecule. Each node is a molecule, and each edge is a reaction transforming products back to reactants. Key properties:

• Root: The target molecule to synthesize
• Leaves: Commercially available building blocks that terminate the search
• Depth: The number of linear steps in the longest path RL agents learn a policy that selects which node to expand and which disconnection to apply, optimizing for tree properties like minimal depth or maximum convergence.

A planning paradigm where the RL reward function incorporates real-world economic factors beyond simple step count:

• Starting material cost: Price per gram from building block catalogs
• Reagent cost: Expense of catalysts, solvents, and coupling agents
• Yield penalties: Cumulative probability of success across linear steps
• Waste disposal: Environmental and regulatory costs The agent learns to prefer routes using inexpensive, in-stock precursors over shorter but prohibitively expensive alternatives. This transforms retrosynthesis from a feasibility problem into an economic optimization problem.

A curated catalog of commercially available or in-stock compounds that serve as terminal nodes in retrosynthetic search. When an RL agent's disconnection produces a molecule found in this library, the branch terminates successfully. Key characteristics:

• Size: Ranges from thousands (in-house collections) to millions (vendor catalogs like eMolecules)
• Availability: Real-time inventory and pricing data
• Filtering: Pre-filtered by lead time, purity, and regulatory status The library defines the action space boundary—the agent cannot plan routes requiring unavailable precursors.

A synthetic strategy where multiple molecular fragments are synthesized independently and then coupled together at a late stage. This is the ideal outcome for RL-based planners because:

• Reduces linear step count: Parallel synthesis shortens the longest path
• Improves overall yield: Fewer sequential steps mean less cumulative yield loss
• Enables modularity: Fragments can be optimized separately RL reward functions heavily incentivize convergent routes by assigning higher scores to trees with lower depth-to-breadth ratios, directly impacting manufacturing efficiency.