Offline Store

Designed to persist petabytes of time-series feature data over months or years. Unlike the online store, which prioritizes speed, the offline store prioritizes storage density and cost-efficiency. It retains a complete, append-only log of all feature values, enabling the reconstruction of any historical state. This is critical for training models on long-tail events and seasonal trends that are invisible in short-term windows. Common physical implementations include cloud data lakes like Amazon S3 or distributed file systems like HDFS, often using columnar formats such as Parquet to optimize analytical queries.

The offline store is built to serve as a source for distributed compute engines like Apache Spark, Snowflake, or BigQuery. These engines execute large-scale feature engineering jobs directly against the stored data. Key operations include:

• Backfilling: Populating a new feature definition across years of historical data.
• Time Travel: Querying a feature's state at any arbitrary point in the past.
• Aggregation: Computing rolling window statistics (e.g., 30-day average purchase value) over massive entity sets. This architecture decouples heavy computation from the low-latency online store, ensuring training workloads never degrade real-time serving performance.

The primary output of the offline store is a feature vector dataset for model training. The process involves joining multiple feature groups on a common entity (like a user ID) and a timestamp. The result is a flat, denormalized table where each row is a complete training example. This table is typically materialized into a file format like TFRecord or Parquet and consumed by frameworks like TensorFlow or PyTorch. The offline store ensures that the exact same feature transformation logic used for training can be replicated in the online store for inference, enforcing training-serving skew prevention.

The offline store acts as the physical backing for the feature registry. Data scientists can query the store's metadata and sample data to discover existing features before engineering new ones. This promotes feature reuse across teams and models. By browsing the offline store's catalog, a user can find pre-computed features like 'customer_lifetime_value_90d' with full feature lineage—tracing it back to the raw source tables and transformation code. This reduces duplicate engineering effort and ensures consistent feature definitions across an organization.

The offline store serves as the source of truth from which data is materialized into the online store. A scheduled batch job reads the latest feature values from the offline store and pushes them to the low-latency online database (e.g., Redis or DynamoDB). This ensures that the online store contains a consistent, up-to-date snapshot for real-time serving. The offline store retains the full history, while the online store holds only the most recent value per entity, optimizing for read speed during prediction requests.

A core data engineering guarantee that the offline store must enforce. When generating a training dataset, feature values must be reconstructed exactly as they existed at the historical timestamp of the event, not using current values. This prevents data leakage from the future into the training set. The offline store's ability to perform time travel queries is the mechanism that enables this correctness.

The process of pre-computing feature values and persisting them from the offline store into the online store. This is typically a scheduled batch job that reads the latest feature values from the offline store's batch sources and upserts them into the low-latency online database. Without this step, the online store would have no data to serve for real-time predictions.

Feature values computed on static, historical datasets using distributed processing frameworks like Apache Spark or Snowflake. These are the primary type of data stored in the offline store. They capture long-term trends and aggregations, such as:

• A customer's average spend over the last 90 days
• The total number of items sold in a region last quarter

A logical abstraction that defines how raw data is transformed into a feature set. A feature view specifies the join logic between entities and feature groups, and it is the object from which a training dataset is generated. When a data scientist requests a training dataset from the offline store, they are querying a specific feature view, which the store resolves into a point-in-time correct join.

The process of computing and populating feature values for a historical time range. When a new feature is defined, it has no historical values. Backfilling runs a batch computation over the offline store's source data to generate feature values for the past, enabling the feature to be used in training datasets that include historical events. This is a heavy, high-throughput operation perfectly suited for the offline store's scalable compute engine.