Kalman Filter

The Kalman filter operates through a continuous two-step recursive loop. In the prediction step, the filter projects the current state estimate and its uncertainty forward to the next time step using a process model. In the update (correction) step, it incorporates a new noisy measurement, weighting it against the prediction based on their relative uncertainties via the Kalman gain. This gain dynamically balances trust between the model's forecast and the actual observation, making the filter optimal for linear Gaussian systems.

The Kalman filter is the minimum mean-square error (MMSE) estimator for linear dynamic systems with additive white Gaussian noise. It assumes that both the process noise (model uncertainty) and measurement noise (sensor or data noise) follow zero-mean Gaussian distributions. Under these conditions, no other algorithm can produce a more accurate state estimate. For financial transaction monitoring, this means the filter provides the statistically optimal baseline for expected spending behavior when the underlying dynamics are approximately linear.

The filter models a system using a state space formulation with two equations:

• State transition equation: x_k = F * x_{k-1} + w_k — describes how the hidden state evolves over time.
• Observation equation: z_k = H * x_k + v_k — maps the hidden state to the observable measurement.

In fraud detection, the state x_k might represent a user's latent spending propensity, while z_k is the observed transaction amount. The matrices F and H define the linear dynamics and measurement mapping respectively.

A defining feature of the Kalman filter is that it maintains and propagates a covariance matrix P alongside the state estimate. This matrix quantifies the uncertainty in every dimension of the estimated state. After each update, the covariance shrinks, reflecting increased confidence. During prediction, it grows, reflecting accumulated uncertainty. This provides a principled, probabilistic anomaly score: if an observed transaction falls far outside the predicted distribution defined by the state and its covariance, it is flagged as suspicious.

The Kalman gain K is the critical computational heart of the filter, calculated as: K = P_pred * H^T * (H * P_pred * H^T + R)^{-1}

It determines the weight given to the new measurement versus the prediction. When measurement noise R is high, the gain is low, and the filter trusts its prediction more. When prediction uncertainty P_pred is high, the gain is high, and the filter rapidly adapts to new measurements. This adaptive behavior is essential for tracking users whose spending patterns gradually shift over time.

Standard Kalman filters assume linear dynamics, which may not hold for complex financial behavior. Two key extensions address this:

• Extended Kalman Filter (EKF): Linearizes the system around the current estimate using Jacobian matrices, suitable for mildly non-linear transaction patterns.
• Unscented Kalman Filter (UKF): Uses a deterministic sampling technique (the unscented transform) to propagate sigma points through the non-linear function, avoiding linearization errors and providing superior accuracy for highly non-linear spending models.

A statistical model representing a system as a Markov process with unobserved (hidden) states. Each hidden state emits an observable output, such as a transaction amount or merchant type. In fraud detection, HMMs model a user's latent behavioral state—like 'normal spending' or 'travel mode'—and flag transactions with low emission probabilities as anomalies. Unlike the Kalman Filter, which operates on continuous state spaces, HMMs use discrete states, making them suitable for categorical sequences like merchant category codes.

A time-series forecasting approach where the prediction for the next time step is a function of its own previous values. The AR(p) model predicts the next transaction amount as a linear combination of the past p observations. This directly parallels the Kalman Filter's prediction step, where the prior state estimate propagates forward. Autoregressive models establish a dynamic baseline for expected transaction amounts, and significant deviations from predictions serve as fraud indicators.

A forecasting method that applies decreasing weights to past observations, with the ETS framework explicitly modeling Error, Trend, and Seasonality components. This is conceptually related to the Kalman Filter's recursive updating, where the Kalman gain determines how much weight to give new measurements versus the prior estimate. ETS provides a lightweight, interpretable baseline for expected transaction behavior, useful when computational simplicity is prioritized over the Kalman Filter's full probabilistic state estimation.

An algorithmic technique for identifying moments in a time series where the statistical properties shift abruptly. While the Kalman Filter assumes smooth state evolution with Gaussian noise, change point detection explicitly models regime changes—such as an account takeover causing a sudden shift in spending patterns. Algorithms like PELT or Bayesian Online Change Point Detection complement the Kalman Filter by segmenting transaction histories into distinct behavioral regimes before applying state estimation.

The phenomenon in streaming data where the underlying statistical relationship between input features and the target variable changes over time. A Kalman Filter's state transition matrix may become miscalibrated as fraudster tactics evolve. Monitoring for concept drift—using methods like ADWIN or DDM—signals when the filter's assumptions no longer hold, triggering model retraining or adaptive parameter updates to maintain detection efficacy against shifting criminal behavior.

A sequential analysis technique that accumulates deviations from a target mean over time, triggering an alert when the cumulative sum exceeds a threshold. CUSUM is highly sensitive to small, sustained shifts—like a fraudster gradually increasing transaction amounts to avoid single-transaction thresholds. This complements the Kalman Filter's residual monitoring: while the filter provides optimal one-step-ahead predictions, CUSUM on the innovation sequence detects persistent biases indicating model failure or evolving fraud patterns.