Backtesting

The foundational principle of backtesting is strict temporal separation. The model must be trained exclusively on data from a historical window and tested on a subsequent, non-overlapping period. This simulates the true production environment where the future is unknown.

• Training Set: Q1 2022 – Q4 2022
• Holdout/Test Set: Q1 2023 – Q2 2023
• Key Metric: Performance delta between in-sample and out-of-sample periods. A significant drop indicates overfitting to historical noise rather than learning generalizable fraud patterns.

Standard k-fold cross-validation leaks future information in time-series data. Purged cross-validation eliminates this leakage by removing training data points that temporally overlap or immediately precede test data points within each fold.

• Purging: Excludes training observations whose event windows overlap with the test set.
• Embargo: Applies a gap period after the test set to prevent training on data that would not have been available in a live deployment.
• This technique is critical for fraud models where a single criminal event can span multiple transaction records across days.

A rigorous backtesting framework must evaluate model performance across combinatorial slices of the data, not just aggregate metrics. This exposes hidden vulnerabilities where a model performs well on average but fails catastrophically for specific segments.

• Slicing Dimensions: Transaction amount bands, merchant category codes, geographic regions, time-of-day windows, and customer tenure cohorts.
• Minimum Cell Size: Each slice must contain a statistically significant number of fraud cases to ensure reliable metric calculation.
• Disparate Impact Check: Identifies slices where the false positive rate is disproportionately high, signaling potential fair lending risk.

Backtesting must simulate performance under distinct behavioral regimes to ensure the model is not brittle to market shifts. A model trained during a low-fraud period may fail when attack velocity spikes.

• Regime Scenarios:
  • Baseline: Normal transaction volume and fraud prevalence.
  • Spike Attack: 10x increase in fraud attempts over 72 hours.
  • Economic Shock: Rapid change in consumer spending patterns.
  • New Product Launch: Introduction of an unseen payment method.
• Expected Output: A regime sensitivity matrix quantifying precision and recall degradation under each scenario.

While Population Stability Index measures input feature drift, backtesting must also quantify prediction score drift. A stable model should produce a consistent distribution of risk scores over time under normal conditions.

• Calculation: Compares the distribution of model scores in the development window against the validation window using a symmetric logarithmic divergence measure.
• Thresholds:
  • PSI < 0.1: No significant shift.
  • 0.1 ≤ PSI < 0.25: Moderate shift requiring investigation.
  • PSI ≥ 0.25: Critical shift indicating potential model breakdown.
• A rising PSI without a corresponding rise in actual fraud is a leading indicator of concept drift.

Backtesting is not a one-time pre-deployment exercise. It must be embedded into the MLOps pipeline as a recurring, automated gate that triggers on both cadence and event-driven conditions.

• Scheduled Cadence: Monthly re-backtesting on the most recent completed data window.
• Event-Driven Triggers:
  • A new champion-challenger candidate is proposed.
  • A significant data drift alert fires.
  • A regulatory finding requires model re-validation.
• Automated Artifact: Each run generates a versioned backtesting report with pass/fail criteria, automatically routed to the Model Risk Management committee.

A symmetric metric that quantifies the shift in a variable's distribution between a development sample and a validation or production sample. PSI serves as a primary red-flag indicator for data drift in ongoing model monitoring. The calculation bins expected and actual frequencies, then sums the logarithmic differences:

• PSI < 0.1: No significant shift
• PSI 0.1–0.25: Moderate shift requiring investigation
• PSI > 0.25: Significant shift demanding model recalibration

A controlled experimentation methodology where a live champion model runs in parallel with one or more challenger models on identical production traffic. This empirically validates that a new model variant outperforms the incumbent before full deployment. The framework enables:

• Head-to-head comparison on live, unbiased data
• Statistical significance testing on performance deltas
• Gradual traffic shifting based on confidence intervals
• Rapid rollback capability if the challenger underperforms

A safe rollout technique where a new model version processes live traffic and logs predictions without impacting actual decisions. This silent validation approach allows teams to:

• Measure latency and resource consumption under real load
• Compare prediction distributions against the production model
• Detect integration issues before customer-facing impact
• Validate monitoring dashboards and alerting thresholds

The simulation of a model's performance under extreme but plausible adverse economic or behavioral scenarios. For fraud detection, this includes testing model stability during:

• Sudden shifts in transaction volume (e.g., holiday spikes)
• Emergence of novel fraud attack vectors
• Coordinated adversarial campaigns targeting specific model weaknesses
• Macroeconomic shocks that alter legitimate spending patterns

A formal, periodic sign-off by accountable business and technology owners confirming that a model remains fit for purpose, compliant with policy, and operating within its defined risk appetite. Attestation typically requires:

• Review of latest monitoring dashboards and drift metrics
• Confirmation of no unapproved changes to model code or data pipelines
• Assessment of any operational incidents or override patterns
• Documentation of any compensating controls for identified weaknesses