Data Preprocessing

Missing values are gaps or null entries in a dataset that can disrupt model training. Techniques include:

• Imputation: Replacing missing values with statistical measures (mean, median, mode) or predicted values from other features.
• Deletion: Removing rows or columns with excessive missing data, though this risks losing valuable information.
• Flagging: Adding a binary indicator feature to signal where data was originally missing, allowing the model to learn from the absence pattern. For multimodal data, missing values in one modality (e.g., a corrupted audio file) may require coordinated handling with its paired data (e.g., the corresponding transcript).

This technique adjusts the range or distribution of numerical features to a standard scale, preventing features with larger magnitudes from dominating the model. Common methods are:

• Standardization (Z-score): Transforms data to have a mean of 0 and a standard deviation of 1. Formula: (x - mean) / std.
• Min-Max Scaling: Rescales data to a fixed range, typically [0, 1]. Formula: (x - min) / (max - min).
• Robust Scaling: Uses the median and interquartile range, making it resistant to outliers. In multimodal contexts, each modality's features (e.g., pixel intensities, audio decibels, word counts) require separate, appropriate scaling before fusion.

Machine learning models require numerical input, so categorical data (text labels, IDs) must be converted. Key methods include:

• One-Hot Encoding: Creates a new binary column for each category. Ideal for nominal data without order (e.g., city names).
• Ordinal Encoding: Assigns a unique integer to each category, preserving a meaningful order if one exists (e.g., 'low', 'medium', 'high').
• Embedding Layers: For high-cardinality categories (e.g., user IDs), a learned dense vector representation is often used within the model itself. For cross-modal tasks, categorical labels (like object classes in an image) must be consistently encoded with their paired text descriptions.

Outliers are data points that significantly deviate from the rest of the distribution and can skew model learning. Detection methods include:

• Statistical Methods: Using Z-scores or Interquartile Range (IQR) to identify points beyond a defined threshold (e.g., >3 standard deviations).
• Visualization: Box plots and scatter plots for manual inspection.
• Model-Based: Isolation Forest or DBSCAN clustering. Treatment involves capping/winsorizing (limiting extreme values), transformation (log scaling), or removal. In sensor fusion, an outlier in one sensor stream must be evaluated in the context of other synchronized modalities.

Data augmentation artificially expands the training dataset by applying label-preserving transformations, improving model generalization and robustness. Modality-specific techniques include:

• Image/Video: Random cropping, rotation, flipping, color jitter, and adding noise.
• Audio: Adding background noise, time stretching, pitch shifting, and speed perturbation.
• Text: Synonym replacement, random insertion/deletion, back-translation. For multimodal augmentation, transformations must be applied consistently across paired data. For example, rotating an image should correspond to spatially adjusting its associated bounding box annotations.

This technique reduces the number of random variables (features) under consideration, combating the 'curse of dimensionality' to improve model efficiency and reduce noise.

• Feature Selection: Choosing a subset of the most relevant features using methods like mutual information or model-based importance scores.
• Feature Extraction: Projecting data into a lower-dimensional space. Principal Component Analysis (PCA) is a linear method, while t-SNE and UMAP are non-linear techniques useful for visualization. In multimodal pipelines, dimensionality reduction is often applied per modality before creating a unified embedding space, ensuring computational tractability.

Text preprocessing converts unstructured language into numerical representations. Tokenization splits text into smaller units (tokens), such as words or subwords. Vectorization then maps these tokens to dense vectors (embeddings) using models like BERT or GPT. Key steps include:

• Lowercasing and removing punctuation for normalization.
• Handling out-of-vocabulary (OOV) tokens with special markers.
• Applying padding or truncation to create uniform sequence lengths for batch processing. For example, the sentence "The model processes data." might be tokenized into ["the", "model", "process", "##es", "data", "."] and then converted into a 768-dimensional vector per token.

Image preprocessing standardizes visual inputs for convolutional neural networks (CNNs). Resizing (e.g., to 224x224 pixels) ensures consistent input dimensions. Normalization scales pixel values, typically to a range of [0,1] or [-1,1], or standardizes using the dataset's mean and standard deviation (e.g., ImageNet stats: mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]). Common operations include:

• Color space conversion (RGB to grayscale).
• Data augmentation like random cropping, flipping, or rotation to increase robustness.
• Channel ordering (e.g., converting HWC to CHW format for PyTorch).

Audio preprocessing transforms raw waveform signals into time-frequency representations. The core step is computing a spectrogram via Short-Time Fourier Transform (STFT), which reveals frequency content over time. Common refinements include:

• Mel-spectrograms: Warping the frequency axis to the Mel scale, which better matches human auditory perception.
• Log-mel spectrograms: Applying a logarithmic compression to amplitude, enhancing quieter sounds.
• MFCCs (Mel-Frequency Cepstral Coefficients): Further compressing the spectrogram to represent the spectral envelope. Preprocessing also involves resampling to a standard rate (e.g., 16kHz), silence trimming, and normalizing amplitude.

Video preprocessing deals with spatial and temporal dimensions. Temporal sampling extracts key frames at a fixed rate (e.g., 1 frame per second) to manage computational load. Spatial processing applies image preprocessing to each frame. For motion-aware models, optical flow is calculated between consecutive frames to capture pixel-wise movement vectors as a separate input channel. Advanced pipelines may also perform:

• Face or object detection to crop regions of interest.
• Temporal chunking into fixed-length clips (e.g., 16-frame segments).
• Frame interpolation to standardize variable source frame rates.

Tabular data preprocessing ensures numerical stability and handles incomplete records. For missing values, common strategies are:

• Imputation: Replacing missing entries with the column's mean, median, or mode.
• Indicator columns: Adding a binary flag to mark which values were imputed. Feature scaling is critical for gradient-based models:
• Standardization (Z-score normalization): (x - mean) / std.
• Min-Max Scaling: (x - min) / (max - min) to a range like [0,1]. Categorical encoding is also required, using one-hot encoding for nominal features or ordinal encoding for features with inherent order.

Multimodal preprocessing synchronizes data streams from different sources. Temporal alignment ensures events across modalities correspond in time. For a video-audio pair, this involves:

• Timestamp synchronization using a common clock or manual annotation.
• Interpolation to align different sampling rates (e.g., 30 fps video with 44.1kHz audio). Cross-modal pairing creates corresponding samples, such as linking an image to its descriptive text caption. This often requires:
• Metadata parsing to find matching IDs.
• Validation to ensure pairs are semantically correct (e.g., the caption accurately describes the image).
• Chunking long sequences (like a lecture) into shorter, aligned segments for training.

Feature engineering is the process of creating new input variables (features) from raw data to improve a model's predictive power. It involves domain knowledge to transform data into formats that better represent the underlying problem to the learning algorithm.

• Examples: Creating polynomial features, aggregating time-series data into rolling averages, or extracting the day of the week from a timestamp.
• Contrast with Preprocessing: While preprocessing cleans and standardizes data, feature engineering creates new, informative representations. It is often the most impactful step for model performance.

Normalization and Standardization are scaling techniques used to bring numerical features onto a common scale, which is critical for algorithms sensitive to feature magnitude, like gradient descent-based models.

• Normalization (Min-Max Scaling): Rescales features to a fixed range, typically [0, 1]. Formula: (x - min(x)) / (max(x) - min(x)). Useful when data bounds are known.
• Standardization (Z-score Normalization): Rescales features to have a mean of 0 and a standard deviation of 1. Formula: (x - mean(x)) / std(x). Less affected by outliers and is the default for many algorithms.

One-hot encoding is a technique for converting categorical variables into a binary vector representation suitable for machine learning models. Each unique category value becomes a new binary feature (column).

• Mechanism: For a categorical feature with n unique categories, n new binary columns are created. A sample is assigned a 1 in the column corresponding to its category and 0 in all others.
• Use Case: Essential for feeding non-ordinal categorical data (e.g., city names, product types) into algorithms that require numerical input. It prevents the model from incorrectly inferring ordinal relationships.

Imputation is the process of replacing missing data values with substituted, statistically plausible values. It is a critical step to handle incomplete datasets without discarding valuable samples.

• Common Techniques:
  • Mean/Median/Mode Imputation: Replaces missing values with the feature's central tendency. Simple but can reduce variance.
  • K-Nearest Neighbors (KNN) Imputation: Uses values from the k most similar samples.
  • Model-Based Imputation: Predicts missing values using a regression or other model trained on the complete features.
• Consideration: The choice of method depends on the data's missingness mechanism (MCAR, MAR, MNAR).

Data augmentation is a set of techniques to artificially expand the size and diversity of a training dataset by applying label-preserving transformations. It is a form of regularization that improves model generalization.

• In Computer Vision: Random rotations, flips, cropping, color jittering, and noise addition.
• In Natural Language Processing: Synonym replacement, random word insertion/deletion/swap, and back-translation.
• For Multimodal Data: Must preserve cross-modal alignment (e.g., augmenting an image and its corresponding text caption in a synchronized manner).

Dimensionality reduction is the transformation of data from a high-dimensional space to a lower-dimensional space, aiming to retain the most important patterns or relationships. It is often used as a preprocessing step.

• Goals: Reduce computational cost, mitigate the "curse of dimensionality," remove noise, and enable visualization.
• Key Algorithms:
  • Principal Component Analysis (PCA): A linear technique that finds orthogonal axes of maximum variance.
  • t-Distributed Stochastic Neighbor Embedding (t-SNE): A non-linear technique primarily for visualization.
  • Uniform Manifold Approximation and Projection (UMAP): A non-linear technique for both visualization and general-purpose reduction.