Molecular Transformer

The core architecture treats molecular informatics as a language translation problem. It uses an encoder-decoder transformer stack to map an input molecular representation (source sequence) to an output representation (target sequence). For example, it can translate a SMILES string into a SELFIES representation, or generate a molecular graph from a linear notation. The encoder processes the source molecule into a continuous latent representation, and the decoder autoregressively generates the target sequence one token at a time, attending to the encoder's output via cross-attention mechanisms.

The transformer's multi-head self-attention mechanism is critical for capturing long-range dependencies within a molecule. Unlike recurrent networks, self-attention allows every atom or token in a SMILES string to directly interact with every other atom or token in a single operation. This enables the model to learn complex, non-local structural relationships:

• Ring closures: Linking distant atoms that form a cycle
• Branching points: Understanding substituent relationships in a linear notation
• Stereochemistry: Capturing chiral centers far apart in the sequence This global receptive field is essential for accurately representing the true topology of a molecule.

Since the transformer has no inherent sense of sequence order, positional encodings are added to the input embeddings. For molecular applications, standard sinusoidal encodings can be augmented or replaced with structural encodings that capture the 2D or 3D topology of the molecule. This can include:

• Graph Laplacian eigenvectors: Encoding the connectivity of the molecular graph
• Shortest-path distances: Representing the topological distance between atom pairs
• 3D conformer coordinates: Injecting spatial information for geometry-aware translation These encodings allow the model to distinguish between structural isomers that have identical atomic compositions but different connectivities.

A critical design choice is the tokenization strategy for molecular strings. The vocabulary can be built at different levels of granularity:

• Atom-level tokens: Each atom symbol, bond type, and ring closure digit is a separate token (e.g., C, =, (, 1)
• Substructure tokens: Common functional groups like C(=O)O (carboxylic acid) are treated as single tokens via Byte-Pair Encoding (BPE)
• Character-level tokens: Each character is a token, providing maximum flexibility but longer sequences Substructure tokenization often improves performance by injecting chemical prior knowledge and reducing sequence length, making the attention computation more efficient.

The encoder component is often pre-trained using a masked language modeling (MLM) objective, similar to BERT. During pre-training, random tokens in a SMILES string are masked, and the model learns to predict them from the surrounding bidirectional context. This forces the encoder to build a deep, contextualized understanding of molecular syntax and semantics. The pre-trained encoder can then be fine-tuned for downstream tasks like property prediction or used as the initialization for the full encoder-decoder translation model, significantly improving sample efficiency and generalization.

The decoder generates the target molecular representation autoregressively, predicting the next token based on the previously generated tokens and the encoder's output. A key challenge is ensuring the generated sequence corresponds to a valid molecule. Strategies to enforce validity include:

• Grammar-constrained decoding: Masking invalid next-token probabilities based on a molecular syntax grammar (e.g., SMILES grammar)
• SELFIES as target representation: Since every SELFIES string is a valid molecule, using it as the target language guarantees 100% validity
• Beam search with chemical rules: Scoring and filtering candidate sequences based on valence rules and ring closure consistency

A machine learning paradigm where a single model is trained simultaneously on multiple related biological or physicochemical endpoints. For Molecular Transformers, this approach allows the model to learn shared representations across tasks such as solubility prediction, toxicity assessment, and bioactivity estimation. By leveraging common underlying chemical features, multi-task architectures improve generalization performance and data efficiency, particularly for endpoints with limited training data.

The process of assigning a confidence interval or probability distribution to a model's prediction, distinguishing between:

• Aleatoric uncertainty: Inherent noise in the data
• Epistemic uncertainty: Model ignorance due to limited training data

For Molecular Transformers applied to property prediction, uncertainty quantification is essential for identifying molecules outside the applicability domain and prioritizing compounds for experimental validation.

SHapley Additive exPlanations provide a game-theoretic approach to explaining model outputs by computing the marginal contribution of each molecular feature to a prediction. When applied to Molecular Transformer outputs, SHAP values can highlight which substructural fragments or atoms most influenced a predicted property, offering medicinal chemists interpretable insights into structure-activity relationships and building trust in black-box predictions.