Threshold Moving

Most classifiers use a default decision threshold of 0.5, meaning any instance with a predicted probability ≥ 0.5 is classified as positive. In imbalanced fraud datasets where the minority class may represent less than 1% of transactions, this default is almost always suboptimal. Threshold moving systematically searches for the probability cutoff that maximizes a chosen metric—such as F1-score, recall, or precision—on a held-out validation set. For example, a fraud model might achieve optimal recall at a threshold of 0.12 rather than 0.50, dramatically increasing true positive detection while accepting a controlled increase in false positives.

The optimal threshold depends entirely on the business objective and the chosen evaluation metric:

• Maximize F1-score: Balances precision and recall, suitable when both false positives and false negatives carry significant cost.
• Maximize recall: Prioritizes catching as many fraud cases as possible, appropriate when missing fraud is far more expensive than investigating false alarms.
• Maximize precision: Minimizes false positives, used when investigation resources are severely constrained.
• Maximize Cohen's Kappa or Matthews Correlation Coefficient: Seeks a threshold that performs well across all four confusion matrix quadrants. The threshold is selected by plotting the metric against candidate probability values using the validation set and choosing the peak.

When the relative cost of errors is known, the optimal threshold can be derived analytically rather than empirically. Given a cost matrix where:

• C_FN: Cost of a false negative (missed fraud)
• C_FP: Cost of a false positive (incorrectly blocked legitimate transaction) The optimal threshold is calculated as: C_FP / (C_FN + C_FP). For example, if a missed fraud costs $5,000 and a false positive investigation costs $50, the optimal threshold becomes 50 / (5000 + 50) ≈ 0.0099. This cost-sensitive approach directly ties the decision boundary to financial impact, making it highly interpretable for risk managers and compliance officers.

Threshold moving offers distinct advantages over data-level resampling techniques like SMOTE or random undersampling:

• No data modification: The original class distribution is preserved, avoiding the introduction of synthetic samples that may not reflect real transaction patterns.
• Computational efficiency: No additional training time or memory overhead; the same trained model is evaluated at multiple thresholds.
• Preserves probability estimates: Resampling distorts the model's internal probability calibration, while threshold moving works with the original output distribution. However, threshold moving does not address fundamental model bias caused by extreme imbalance during training. It is often most effective when combined with cost-sensitive learning or class-weighted loss functions applied during the training phase.

Selecting the threshold on the same data used to evaluate the model introduces optimistic bias and leads to poor generalization. Best practices include:

• Stratified K-Fold: Preserve the minority class proportion in each fold, select the threshold on training folds, and evaluate on the held-out fold.
• Dedicated threshold set: Split data into training (60%), threshold selection (20%), and final evaluation (20%) sets to prevent information leakage.
• Temporal validation: For fraud detection, use time-based splits where the threshold is selected on historical data and validated on future periods to simulate production deployment.
• Confidence intervals: Bootstrap the threshold selection process to produce confidence intervals, ensuring the chosen threshold is stable and not an artifact of a particular validation split.