Data Imputation

These foundational techniques replace missing values with a single constant or central tendency measure. While computationally cheap, they artificially reduce variance and can distort the underlying data distribution.

• Mean/Median Imputation: Replaces nulls with the column's average or median. Best for normally distributed data but sensitive to outliers.
• Mode Imputation: Used for categorical variables, filling gaps with the most frequent category.
• Forward/Backward Fill: Propagates the last valid observation forward. Essential for time-series data where temporal locality implies similarity.
• Constant Value Imputation: Fills missing entries with a fixed scalar (e.g., 0 or 'Unknown'), explicitly flagging the absence of information.

MICE, also known as Fully Conditional Specification, performs multiple regressions iteratively. It predicts missing values for each feature using all other features as predictors, cycling through variables until convergence.

• Iterative Process: Each incomplete variable is regressed on the others, with the process repeating for 10-20 cycles.
• Multiple Imputation: Generates several complete datasets to capture the uncertainty inherent in the imputation process.
• Pooling: Results from multiple imputed datasets are combined using Rubin's Rules to produce final estimates and standard errors.
• Assumption: Relies on the Missing At Random (MAR) assumption, where the probability of missingness depends only on observed data.

KNN imputation identifies the k most similar complete records using a distance metric and averages their values to fill the gap. It preserves local structure but is sensitive to the curse of dimensionality.

• Distance Metrics: Typically uses Euclidean or Manhattan distance on normalized feature spaces.
• Heterogeneous Data: A variant using Gower's distance can handle mixed numerical and categorical data types simultaneously.
• Weighted KNN: Assigns higher weights to closer neighbors, often using an inverse distance weighting scheme.
• Limitations: Computationally expensive at inference time and requires careful selection of k to avoid over-smoothing.

Advanced techniques treat the dataset as a matrix and decompose it into latent factors, reconstructing missing entries. These methods excel at capturing complex, non-linear interactions in high-dimensional alternative data.

• Matrix Factorization: Techniques like Singular Value Decomposition (SVD) or Non-negative Matrix Factorization learn low-rank representations to predict missing cells.
• Autoencoders: Neural networks trained to reconstruct their input. A Denoising Autoencoder is specifically trained to recover original values from artificially corrupted data.
• Generative Adversarial Networks (GAIN): Uses a generator to impute values and a discriminator to distinguish real from imputed data, producing highly realistic synthetic values.
• MissForest: An iterative random forest method that bootstraps predictions, handling non-linear relationships and interactions without parametric assumptions.

Financial tick data and sensor streams require specialized methods that respect temporal ordering and autocorrelation. Standard cross-sectional imputation breaks the sequential dependency critical for forecasting.

• Interpolation: Linear, spline, or polynomial methods estimate values between known timestamps. Cubic spline provides smoothness for pricing curves.
• Kalman Smoothing: A recursive Bayesian filter that estimates the true state of a noisy system, providing optimal imputation for linear Gaussian state-space models.
• Seasonal Decomposition: Imputes missing values by decomposing the series into trend, seasonal, and residual components using STL or LOESS.
• Last Observation Carried Forward (LOCF): A conservative medical and financial standard that assumes no change until evidence appears, preventing forward-looking bias.

Validating imputation quality is critical to prevent garbage-in, garbage-out scenarios in quantitative models. Diagnostics compare the distribution of imputed versus observed values.

• Density Plots: Overlay kernel density estimates of original and imputed data to visually check for distributional shift.
• Cross-Validation: Artificially introduce missingness (MCAR or MAR) into complete data, impute, and measure RMSE against known true values.
• Coverage Probability: In multiple imputation, checks if the confidence intervals contain the true parameter at the nominal rate.
• Imputation Uncertainty: Always report the fraction of missing information (FMI) to quantify the added variance from the imputation process.