Batch Features

Batch features are derived from offline data warehouses or data lakes containing snapshots of historical records. Unlike streaming features, they do not process real-time events. The computation runs against a fixed, bounded dataset—for example, all user transactions from the previous three years. This allows for complex, multi-stage aggregation pipelines that would be impossible to execute with low-latency constraints.

• Source: Data warehouses (Snowflake, BigQuery), data lakes (S3, ADLS)
• Cadence: Typically daily or hourly, not sub-second
• Example: A user's total spend over the last 90 days, computed at 2 AM UTC

The primary purpose of batch features is to encode slow-moving signals and historical patterns. These features provide the model with context that cannot be inferred from a single real-time event. Common aggregations include rolling window statistics, lifetime counts, and decay-weighted averages.

• Rolling Window: Average order value over the last 30 days
• Lifetime Metrics: Total number of purchases since account creation
• Cohort Analysis: A user's percentile rank within their acquisition cohort
• Seasonality: Purchase frequency during the same calendar month last year

Due to the massive scale of historical data, batch feature computation relies on distributed processing engines that parallelize workloads across clusters. These frameworks handle data shuffling, partitioning, and fault tolerance to produce feature values efficiently.

• Apache Spark: The dominant framework for batch feature engineering, using DataFrames and SQL for transformations
• Apache Hive: SQL-based processing on Hadoop for legacy pipelines
• dbt: For transformation logic in modern data warehouses, often used for simpler aggregations
• Execution Pattern: MapReduce-style operations that scan, filter, group, and aggregate entire datasets

Once computed, batch features are persisted in the feature store's offline store—a high-throughput, scalable storage layer optimized for large-scale retrieval during model training. The offline store acts as the system of record for historical feature values.

• Storage Backends: Parquet files on object storage, Delta Lake tables, or cloud data warehouses
• Retrieval Pattern: Full table scans for training dataset generation, not point lookups
• Point-in-Time Correctness: The offline store must support time-travel queries to reconstruct feature values exactly as they existed at any historical timestamp, preventing data leakage from future information

Although computed in batch, these features are often materialized into the online store to serve low-latency predictions. The materialization process copies the latest computed values into a key-value database optimized for point lookups during inference.

• Materialization Job: A scheduled process that reads the latest batch values and upserts them into the online store
• Latency Trade-off: Batch features in the online store have T+1 freshness—they reflect the state as of the last batch run, not the current moment
• Use Case: A user's 'favorite category' computed nightly and served instantly at login the next morning

Batch features and streaming features serve complementary roles in a feature store. Batch features provide deep historical context, while streaming features capture immediate intent. Understanding the distinction is critical for feature selection.

• Batch: User's 90-day average cart size (computed nightly)
• Streaming: Items in the user's current session cart (updated in real-time)
• Hybrid Use: A model often combines both—using batch features for long-term preferences and streaming features for session context
• Infrastructure: Batch uses Spark on offline data; streaming uses Flink or Kafka Streams on event logs