Data Leakage

The most insidious form of leakage in A/B testing infrastructure occurs when future information is inadvertently used to predict past events.

• Mechanism: Training a model on features that were not yet available at the time of prediction, such as using post-purchase behavior to predict purchase intent.
• Example: A churn model trained on customer service tickets filed after the cancellation event, achieving 99% AUC offline but 50% in production.
• Detection: Strictly timestamp all features and validate that no feature's observation time exceeds the label's timestamp during train/test splitting.

A fundamental data processing error where statistics computed from the entire dataset are applied before splitting, allowing the test set to influence the training process.

• Common Pitfall: Applying normalization, standardization, or missing value imputation using the global mean and standard deviation before performing the train/test split.
• Impact: The model 'sees' the scale and distribution of the test data during training, artificially inflating performance metrics.
• Fix: Fit scalers and imputers exclusively on the training split, then transform the validation and test sets using those learned parameters.

Occurs when a feature is not a direct copy of the label but serves as a highly correlated proxy that would not be available in a real-time inference scenario.

• Example: Using 'number of support tickets filed' as a feature to predict customer churn. The act of filing a ticket is often a consequence of the decision to churn, not a predictor.
• A/B Testing Risk: A personalization model using 'time spent on checkout page' to predict conversion will show stellar results offline, but in a live A/B test, the model fails because it cannot observe this feature until the user is already converting.
• Mitigation: Conduct rigorous feature audits with domain experts to classify features as 'pre-decision' or 'post-decision' relative to the target event.

A subtle violation where samples from the same logical group (user, session, device) appear in both training and test sets, causing the model to memorize specific entities rather than learn generalizable patterns.

• Scenario: Splitting a clickstream dataset randomly at the event level, causing events from the same user session to appear in both training and test sets.
• Consequence: The model learns to recognize high-value users rather than the behavioral signals preceding a click, leading to severe overestimation of click-through rate prediction accuracy.
• Solution: Implement group-aware splitting using a hashed user ID or session ID as the stratification key to ensure all events from a single entity reside entirely in one split.

A specific statistical leakage that arises when target-based categorical encodings are calculated using the entire dataset, embedding the label distribution directly into the features.

• Mechanism: Replacing a categorical value like 'product category' with the mean conversion rate for that category, computed across both training and test data.
• Result: The feature becomes a near-perfect predictor, as it contains a smoothed version of the label itself. This is particularly dangerous in high-cardinality categorical features.
• Prevention: Use cross-validated target encoding within the training folds only, or apply additive smoothing with a strong prior to prevent the encoding from collapsing to the raw target mean.

Leakage often originates not in the model code but in the orchestration layer of the MLOps pipeline, where incorrect merge operations silently corrupt the dataset.

• Common Bug: An incorrect JOIN key in a feature store query that duplicates rows and broadcasts label information across unrelated samples.
• Production Impact: A streaming pipeline that joins real-time click data with a batch-exported label table using a non-unique key, causing label information to leak into the feature vectors of subsequent events.
• Guardrail: Implement data validation tests that assert row counts remain invariant after feature joins and that the entropy of the label distribution does not artificially decrease after feature engineering.

The foundational technique for preventing leakage by partitioning a dataset into mutually exclusive subsets before any analysis begins.

• Training Set: Used to fit model parameters
• Validation Set: Used for hyperparameter tuning
• Test Set: Held out until final evaluation

Proper splitting ensures the model is evaluated on truly unseen data. A common pitfall is splitting after normalization or imputation, which leaks aggregate statistics from the test set into the training process.

A strict boundary in time-series modeling where all training data must precede all validation and test data chronologically.

• Prevents look-ahead bias where future information contaminates past predictions
• Essential for financial forecasting, demand prediction, and user behavior modeling
• Violation example: training a churn predictor on customer service interactions that occurred after the churn event

This is the single most common leakage vector in production ML systems.

The alignment between when a feature value was computed and when it would have been available at prediction time in a production setting.

• Online-offline skew: Features computed with batch latency that wouldn't exist in real-time serving
• Label leakage: Including features derived from the target variable itself (e.g., using 'days since last purchase' when the label is 'purchased today')
• Requires strict point-in-time correctness validation in feature stores

A subtle failure mode where preprocessing steps are applied to the entire dataset before cross-validation folds are created.

• Correct approach: Fit scalers, imputers, and encoders within each fold on training data only
• Incorrect approach: Standardizing the full dataset, then running cross-validation
• This causes optimistically biased performance estimates because information from validation folds bleeds into training folds

Scikit-learn's Pipeline with cross_val_score prevents this automatically.

Occurs when observations from the same logical group (user, device, session) appear in both training and validation sets.

• A single user's multiple transactions split across train/test creates identity leakage
• The model memorizes user-specific patterns rather than learning generalizable behavior
• Solution: Group-aware splitting using GroupKFold or stratified sampling by entity ID
• Critical in recommendation systems where user embedding contamination inflates recall metrics by 20-40%

A leakage vector specific to categorical feature encoding where the target variable's aggregate statistics are computed using the entire dataset.

• Mean target encoding replaces categories with the average target value for that category
• If computed globally, the encoding for each row contains information about its own label
• Fix: Use leave-one-out encoding or compute target statistics strictly within cross-validation folds
• Libraries like category_encoders provide built-in leakage-safe implementations