Data Leakage

Occurs when drugs sharing the same protein target are split across training and test sets. The model memorizes target-specific features rather than learning generalizable drug-target interaction patterns.

• Example: Splitting kinase inhibitors randomly rather than by kinase subfamily
• Consequence: Model appears to predict novel kinase-drug interactions but has simply learned kinase family fingerprints
• Mitigation: Cluster drugs by target class before splitting; use target-stratified cross-validation

Arises when molecules with identical core chemical scaffolds appear in both training and test partitions. The model exploits structural similarity rather than learning true pharmacological relationships.

• Detection: Compute Tanimoto similarity between all training and test molecules; flag pairs above 0.85 threshold
• Standard practice: Use scaffold split where molecules sharing the same Bemis-Murcko scaffold are assigned to the same partition
• Impact: Without scaffold splitting, reported AUC values can be inflated by 10-30%

Occurs when future information contaminates training data in time-series predictions. In drug repurposing, this happens when clinical trial outcomes or post-market surveillance data predate the model's supposed knowledge cutoff.

• Example: Training on adverse event reports from 2023 to predict drug safety in a 2020 validation set
• Real-world failure: Models predicting drug-disease associations using literature published after the hypothetical discovery date
• Solution: Implement time-aware splitting where all training data predates test data by a defined temporal gap

Results from computing dataset-level statistics before splitting, allowing test set properties to influence training normalization or feature selection.

• Common mistake: Applying global feature scaling or principal component analysis to the entire dataset before train-test split
• Example: Normalizing gene expression values using mean and standard deviation computed from combined training and test samples
• Best practice: Fit all preprocessing transformations exclusively on training data, then apply to test set

Occurs when identical or near-identical data points exist in both training and test sets, often due to data curation errors or multiple database entries for the same entity.

• Source: Merging DrugBank and ChEMBL entries without deduplication can create duplicate drug-target pairs
• Detection: Perform exact and fuzzy matching on SMILES strings and target sequences before splitting
• Severity: Even a 1% duplicate rate can artificially boost test accuracy by 5-15% in imbalanced datasets

Arises when proxy variables in the feature set encode information about the test labels. These features act as shortcuts that bypass genuine learning.

• Example: Including 'number of publications' as a feature when predicting drug-disease associations, which correlates with well-studied indications
• Red flag: Features with unusually high feature importance scores that lack mechanistic plausibility
• Audit: Review top SHAP value features for each prediction; question any feature that directly encodes the outcome variable

The most common source of leakage occurs when training and test sets are not properly separated before preprocessing. In drug repurposing, this happens when molecules sharing the same scaffold or Murcko framework are randomly split, placing nearly identical compounds in both sets. The model memorizes the scaffold rather than learning generalizable pharmacological properties.

• Scaffold split is the mandatory alternative to random splitting
• Leakage inflates AUC-ROC by 10-30% in published benchmarks
• Always split at the drug level, not the interaction level

A subtle form of leakage where future information inadvertently informs the training process. In drug repurposing, this occurs when a model is trained on drug-disease associations discovered after the test set's cutoff date. The model appears to 'predict' a known association, but it has simply memorized post-hoc literature.

• Use time-stratified splits based on publication or approval dates
• Critical for Real-World Evidence (RWE) mining from EHRs
• Simulates true prospective validation

Occurs when protein target features derived from the test set's drug-target interactions are used during training. For example, computing protein embeddings or network propagation scores on the full interactome before splitting contaminates the training data with test-set binding information.

• Compute all network features exclusively on the training partition
• Recompute embeddings for each cross-validation fold
• Particularly dangerous in Knowledge Graph Embedding models

In drug repurposing, negative examples (non-interactions) are rarely verified experimentally. Randomly sampling 'negatives' from unlabeled pairs can introduce leakage if the sampling strategy inadvertently selects true but undiscovered interactions. This creates label noise that artificially deflates performance estimates.

• Use PU learning (Positive-Unlabeled) frameworks
• Validate negatives against DrugBank or ChEMBL
• Consider calibrated negative sampling from dissimilar pairs

Many published drug repurposing benchmarks suffer from systemic leakage due to redundant molecular representations. When ECFP4 fingerprints or pre-trained BERT for SMILES embeddings are generated on the full dataset before splitting, the model gains indirect access to test-set structural information through the featurization process.

• Always featurize after splitting, not before
• Audit benchmarks using scaffold similarity heatmaps
• Reproduce baselines with corrected splits to measure true leakage impact