Data Normalization

Transforms features to have a mean of 0 and standard deviation of 1 by subtracting the population mean and dividing by the standard deviation. This technique is essential when algorithms assume normally distributed data, such as linear regression or logistic regression.

• Formula: z = (x - μ) / σ
• Preserves the shape of the original distribution
• Sensitive to outliers, which can distort the mean and standard deviation
• Ideal for features where the actual minimum and maximum are unknown

Rescales features to a fixed range, typically [0, 1] or [-1, 1], by applying a linear transformation based on the minimum and maximum values of the dataset. This is critical for algorithms that require bounded inputs, such as neural networks with sigmoid activation functions.

• Formula: x_scaled = (x - x_min) / (x_max - x_min)
• Preserves zero values and maintains the original distribution's shape
• Highly sensitive to outliers, which compress the scaled range of inliers
• Commonly used in image processing where pixel intensities are bounded

Uses the median and interquartile range (IQR) instead of the mean and standard deviation, making it resilient to outliers. This method centers data around the median and scales according to the IQR, which represents the middle 50% of values.

• Formula: x_scaled = (x - median) / IQR
• IQR is the range between the 25th and 75th percentiles
• Essential for financial datasets with extreme events or fat-tailed distributions
• Does not compress inliers into a narrow range when outliers are present

Normalizes data by moving the decimal point of values based on the maximum absolute value of the feature. The number of decimal places shifted is determined by the largest exponent of 10 that bounds the data.

• Formula: x_scaled = x / 10^j, where j is the smallest integer such that max(|x_scaled|) < 1
• Simple to implement and interpret
• Preserves the relative magnitude between data points
• Less common in modern machine learning pipelines but useful for quick exploratory analysis

Scales individual samples to have a unit norm (length of 1), effectively projecting each data point onto the surface of a hypersphere. This technique is critical when the direction of a vector matters more than its magnitude, such as in text classification with TF-IDF features.

• Common norms: L1 (Manhattan), L2 (Euclidean), and L∞ (Max)
• L2 normalization: x_scaled = x / ||x||₂
• Essential for cosine similarity calculations in information retrieval
• Mitigates the impact of document length in natural language processing tasks

Maps features to follow a uniform or normal distribution by applying a non-linear, rank-based transformation. This method spreads out the most frequent values and reduces the impact of marginal outliers, making it a powerful tool for handling arbitrary distributions.

• Uses the empirical cumulative distribution function (CDF) of the data
• Can force a non-Gaussian feature into a Gaussian-like shape
• Fully non-parametric; makes no assumptions about the original distribution
• Useful for algorithms sensitive to feature distributions, like Gaussian Naive Bayes

The computational process of identifying and merging disparate records that refer to the same real-world entity. Normalization is impossible without first resolving that 'AAPL US' and 'Apple Inc' are identical.

• Uses fuzzy matching and probabilistic linkage algorithms
• Essential for unifying vendor data, SEC filings, and news sentiment
• Poor entity resolution leads to fragmented, duplicate feature sets

The ability to automatically adapt a data system's structure to handle changes in incoming data formats. A robust normalization pipeline must gracefully handle schema drift without breaking downstream models.

• Implements backward-compatible and forward-compatible parsing
• Uses schema registries to version data contracts
• Prevents silent data corruption when vendors change their APIs

The statistical process of replacing missing values with substituted estimates. Normalization strategies must define imputation logic to handle gaps in alternative datasets like satellite imagery or credit card panels.

• Techniques include mean/median substitution, LOCF, and MICE
• Must be fit on training data only to avoid leakage
• Imputation indicators should be preserved as binary feature flags

The precise synchronization of disparate time series to a common, point-in-time index. Normalization is the prerequisite step that enables alignment by converting all sources to a unified timestamp format and frequency.

• Handles timezone conversion, daylight savings, and exchange calendars
• Aligns quarterly macro data with millisecond-level tick data
• Prevents peeking into future information during model training

A centralized platform for storing, versioning, and serving curated feature data. The feature store consumes normalized data and ensures consistency between training and inference environments.

• Provides point-in-time correct feature retrieval APIs
• Manages online (low-latency) and offline (high-throughput) serving
• Enforces the exact same normalization logic in production as in research