Online Feature Serving

Online feature serving must retrieve features in single-digit milliseconds to meet the demands of real-time personalization. This is achieved through in-memory caches, key-value stores like Redis, and direct service-to-service communication that bypasses slow disk I/O. The target is typically p99 latency under 10ms for a full feature vector lookup, ensuring the feature retrieval step does not become the bottleneck in the overall inference pipeline.

Online serving is not a single lookup; it's a real-time join across disparate data sources. A single prediction request may require:

• Pre-computed features from a low-latency online store (e.g., user's 30-day average order value)
• Real-time features computed on the fly from a stream processor (e.g., items viewed in the current session)
• Contextual features passed directly in the request (e.g., device type, current time) The serving layer assembles these into a single, ordered feature vector matching the model's exact input signature.

The logic used to compute a feature online must be byte-for-byte identical to the logic used during training. This requires a shared transformation library or DSL that can execute in both batch (Spark, SQL) and real-time (Flink, Kafka Streams) contexts. Any divergence—such as a different rounding function or null-handling strategy—introduces training-serving skew that silently degrades model performance in production without triggering obvious errors.

Online feature serving is a critical path dependency for real-time applications. If the feature store is unavailable, the model cannot make predictions. Architectures must include:

• Multi-zone replication of the online store
• Circuit breakers that fail open with default feature values
• Local caching on the inference pod to survive network partitions A common pattern is a two-tier cache: a local in-process cache backed by a distributed Redis cluster, providing both speed and resilience.

Online serving must balance data freshness against consistency guarantees. A feature computed from a streaming pipeline may be milliseconds stale but eventually consistent, while a feature from a transactional database can be strongly consistent but slower. Architectures often use a dual-read pattern: serve from the fast, eventually-consistent cache for most requests, but fall back to the authoritative source for high-stakes decisions where staleness is unacceptable.

The silent killer of production ML performance. This occurs when the feature computation logic used during model training diverges from the logic used during online inference. Common causes include:

• Different code paths for batch vs. real-time pipelines
• Stale aggregation windows in the online store
• Missing values handled inconsistently

A feature store eliminates skew by ensuring the same transformation code runs in both environments, guaranteeing point-in-time correctness.

The requirement that features used for training must reflect the exact state of the world as it existed at the time of the prediction, not after the label was observed. Without this:

• Data Leakage: Future information contaminates training data
• Overly Optimistic Metrics: Model appears better in offline evaluation than in production

Achieved through time-travel queries that join features and labels using precise timestamps, ensuring the feature value used is the one that would have been available at prediction time.

The process of transforming raw data into numerical representations that machine learning models can consume. In online serving, features fall into categories:

• Batch Features: Pre-computed aggregates updated hourly/daily (e.g., user lifetime value)
• Real-Time Features: Computed on-the-fly from event streams (e.g., items viewed in current session)
• Context Features: Request-time parameters (e.g., device type, time of day)

Effective feature engineering requires deep domain expertise and is often the highest-ROI activity in ML projects.

Dense, low-dimensional numerical representations of high-dimensional categorical or unstructured data. In feature serving, embeddings are critical for:

• User Embeddings: Capturing behavioral preferences from interaction history
• Item Embeddings: Representing products or content in a semantic space
• Cross-Modal Embeddings: Aligning text, image, and behavioral data

Served from specialized vector databases or in-memory caches, embeddings enable fast approximate nearest neighbor (ANN) search for retrieval and ranking.

A measure of how up-to-date a feature value is at the moment of inference. Stale features degrade model accuracy. Freshness is governed by:

• Materialization Frequency: How often batch features are recomputed
• Stream Processing Lag: End-to-end latency from event ingestion to feature update
• Cache TTL: Time-to-live settings on online store entries

High freshness requires streaming pipelines and write-behind caching to ensure the online store reflects the latest user behavior within milliseconds.