Delayed Feedback

In standard binary classification for CTR/CVR prediction, training instances are labeled immediately. With delayed feedback, a click that will eventually lead to a conversion is temporarily labeled as 0 (negative). This introduces systematic label noise.

• Impact: The model learns to underestimate the true conversion probability.
• Mechanism: The loss function penalizes the model for correctly predicting a high probability on a positive example that hasn't been labeled yet.
• Scale: In long-cycle products (e.g., auto insurance, travel), the delay window can exceed 30 days, meaning a significant fraction of training data is mislabeled at any given time.

The observation window is the fixed period after a click during which a conversion label is expected to arrive. Data is considered censored if the window has not yet elapsed.

• Fixed Window: A click is only labeled negative if no conversion occurs within W days (e.g., 7 days).
• Censored Instances: Clicks that are younger than W days are either excluded from training or modeled with a survival analysis approach.
• Trade-off: A longer window reduces false negatives but increases the staleness of the training data, as you must wait W days to finalize labels.

Specialized architectures that jointly model two probabilities: the conversion probability and the expected delay time.

• Chapelle's Model (2014): A seminal approach that assumes delay follows an exponential distribution and trains a model to predict both P(conversion) and E[delay].
• NoDelay (2020): A neural network approach that uses a survival analysis framework, treating the delay as a censored time-to-event problem.
• ES-DFM (2021): An importance-sampling method that re-weights recent, unlabeled data to correct for the false-negative bias without waiting for the full observation window.

A statistical technique to re-weight training instances based on the probability that their true label has been observed.

• Inverse Propensity Weighting: Each instance is weighted by 1 / P(label observed | elapsed time).
• Fake Negative Weighting: The model assigns a small, non-zero weight to the possibility that a currently negative instance is actually a fake negative.
• Benefit: Allows the model to train on fresh, recent data immediately without waiting for the full observation window, dramatically reducing model staleness.

Delayed feedback doesn't just affect offline metrics; it creates a self-reinforcing bias in production systems.

• Downward Spiral: A model trained on false negatives underestimates CTR/CVR, leading to lower bids and reduced exposure for affected items.
• Reduced Opportunity: Lower exposure means fewer clicks and fewer observed conversions, further starving the model of positive labels.
• Mitigation: Requires explicit exploration mechanisms or counterfactual evaluation to break the cycle and collect unbiased data.

Standard metrics like AUC and Log Loss are biased when evaluated on data with unobserved labels.

• Delayed AUC: Only evaluates on instances where the observation window has fully elapsed, ensuring label certainty.
• Expected Calibration Error (ECE): Measures if predicted probabilities match eventual observed frequencies, requiring a waiting period for ground truth.
• Streaming Evaluation: Continuously re-evaluates predictions as labels trickle in, updating metrics retroactively when a previously negative instance converts.