Change Point Detection

The core mechanism involves continuously monitoring a cost function that quantifies the homogeneity of a data segment. A change point is flagged when the cumulative sum (CUSUM) of residuals or a likelihood ratio exceeds a calibrated threshold. Common metrics include the Kullback-Leibler divergence for distributional shifts and Mahalanobis distance for multivariate mean changes. The algorithm must balance sensitivity to genuine structural breaks against robustness to stochastic noise.

The detection strategy is defined by the latency constraint:

• Online (Sequential) Detection: Processes data points as they arrive, minimizing detection delay (average run length to signal). Uses techniques like the Page-Hinkley test or Bayesian change point models with recursive updates. Essential for real-time personalization engines.
• Offline (Retrospective) Detection: Analyzes a complete, fixed-length time series to identify all change points globally. Employs binary segmentation, PELT (Pruned Exact Linear Time), or dynamic programming to optimize segmentation. Used for historical user journey analysis.

Beyond detecting when a change occurred, the algorithm must characterize what changed in the data-generating process:

• Mean Shift: An abrupt jump in the average value, such as a sudden drop in session dwell time.
• Variance Change: A shift in volatility, indicating increased unpredictability in user click frequency.
• Slope Change: A transition in the trend direction, signaling a gradual but fundamental behavioral pivot.
• Spectral Change: A shift in the frequency domain, often detected using wavelet transforms, revealing changes in cyclical user patterns.

A dominant mathematical framework formulates change point detection as a model selection problem. The objective is to minimize a penalized cost function: Σ cost(segment) + β × (number of change points). The penalty term (β) controls segmentation granularity. Common penalties include the Bayesian Information Criterion (BIC) and Minimum Description Length (MDL). The PELT algorithm solves this exactly in linear time under the condition that the cost function is additive and the penalty is linear.

Change point detection is the primary defense against concept drift in production machine learning systems. When a detection algorithm identifies a virtual concept drift (a change in the data distribution P(X) without affecting the decision boundary), it triggers a feature distribution recalibration. A real concept drift (a change in the posterior P(Y|X)) signals that the predictive relationship has fundamentally broken, necessitating immediate online model retraining or a fallback to a heuristic policy.

A probabilistic framework that maintains a run-length posterior distribution—the belief over how many time steps have elapsed since the last change point. At each step, the model computes the predictive probability of the new observation under the current regime. A change point is inferred when the hazard rate (the prior probability of a change) combined with a low predictive likelihood causes the run-length probability mass to collapse to zero. This provides a full posterior over change point locations rather than a single point estimate.

Detecting an abrupt shift in browsing velocity or dwell time allows the decisioning engine to switch strategies mid-session.

• Mechanism: A CUSUM algorithm monitors the cumulative sum of deviations from a baseline click rate. When the sum exceeds a threshold, a change point is flagged.
• Action: The system instantly transitions from a generic discovery layout to a high-touch, assisted-selling interface.
• Example: A user suddenly viewing 5 high-resolution product images in 10 seconds triggers a live chat invitation.

Change point detection on sales velocity time series identifies sudden demand spikes that break historical seasonal patterns.

• Bayesian Online Change Point Detection (BOCPD) computes the posterior probability of a change in the run length of a purchase rate.
• Result: The supply chain system bypasses standard nightly batch forecasts and triggers an immediate reorder or inter-warehouse transfer.
• Key Metric: A shift from a stable 5-unit/hour rate to 50-unit/hour rate is detected within minutes, not hours.

A user's engagement time series often exhibits a 'pre-churn signature'—a distinct change point where session depth and frequency collapse.

• Method: Pruned Exact Linear Time (PELT) algorithm segments the user's historical login frequency to identify the exact date of behavioral regime change.
• Application: Marketing automation uses this precise date to trigger a retention campaign, rather than relying on a lagging 30-day inactivity rule.
• Advantage: Intervention occurs during the critical 24-hour window after the change point, maximizing re-engagement probability.

Change point detection prevents wasted traffic on underperforming variants by identifying when the posterior probability of a variant being best has stabilized or collapsed.

• Sequential Analysis: Instead of fixed-horizon testing, a change point model monitors the running Bayes factor.
• Trigger: If a sudden change point indicates the treatment effect has vanished or reversed, the experiment is automatically halted.
• Business Impact: Reduces the opportunity cost of running losing experiments by dynamically reallocating users to the winning arm.

Sophisticated fraud rings often shift their attack vectors abruptly, creating a statistical change point in transaction feature distributions.

• Multivariate Detection: The system monitors the KL divergence between a reference window of legitimate transactions and a sliding detection window.
• Response: A detected change point in the correlation between transaction velocity and basket size instantly elevates the risk score for the entire session.
• Outcome: The model adapts to the new fraud regime in real-time, blocking transactions that a static model would miss.

Competitor pricing data streams exhibit structural breaks when a rival launches a clearance event or a new pricing strategy.

• Technique: A robust change point model filters out noise from regular price fluctuations to detect a genuine structural break in the competitor's price index.
• Reaction: The internal pricing algorithm acknowledges the new competitive regime and adjusts its own price elasticity parameters accordingly.
• Example: Detecting a competitor's permanent markdown on a specific SKU category triggers a strategic repricing, not a temporary discount war.

The phenomenon where the statistical properties of a target variable change over time. Unlike abrupt change points, concept drift can be gradual, incremental, or recurring. In retail personalization, concept drift occurs when consumer preferences evolve seasonally or after a product launch.

• Virtual drift: Changes in feature distribution P(X) without affecting the decision boundary
• Real drift: Changes in the conditional distribution P(Y|X) that degrade model accuracy
• Detection methods include ADWIN, DDM, and Page-Hinkley test

A machine learning paradigm where models update continuously as data arrives sequentially, adapting to patterns without full retraining. Critical for e-commerce systems that must react to real-time behavioral shifts.

• Processes one observation at a time using stochastic gradient descent
• Enables immediate adaptation after a detected change point
• Contrasts with batch learning, which requires reprocessing entire historical datasets

The algorithmic task of identifying where one user session ends and another begins. Change point detection is applied to inter-request time distributions to find natural breaks in activity.

• Common approach: timeout-based heuristics (e.g., 30 minutes of inactivity)
• Advanced methods: Bayesian change point models on dwell time sequences
• Accurate boundaries improve downstream sequential models like LSTMs and Transformers

A stochastic process modeling the timing of discrete events through a conditional intensity function λ(t). Hawkes processes capture self-exciting behavior where one purchase increases the probability of subsequent purchases.

• Poisson process: Constant intensity, events occur independently
• Hawkes process: Past events temporarily boost future event probability
• Change point detection on intensity functions reveals regime shifts in user engagement

Assigning a probabilistic value to real-time user behavior quantifying purchase likelihood. Change point detection identifies the exact moment intent shifts from browsing to buying intent.

• Features: scroll velocity, product comparison frequency, coupon searches
• A detected change point in browsing velocity can trigger a next-best-action intervention
• Enables just-in-time personalization before session abandonment

Statistical methods for analyzing time-to-event data, such as time until churn or purchase. Change point detection in hazard functions reveals when risk profiles shift dramatically.

• Kaplan-Meier estimator: Non-parametric survival function estimation
• Cox proportional hazards: Models covariate effects on hazard rates
• Detecting structural breaks in survival curves signals underlying behavioral regime changes