Feature Cache

The primary architectural value of a feature cache is its ability to serve feature vectors in sub-millisecond timeframes, bypassing the network hop and query parsing overhead of a remote online store. This is achieved by storing serialized feature vectors in RAM using systems like Redis or Memcached. For high-throughput inference endpoints processing thousands of queries per second, this latency reduction is critical to maintaining Service Level Objectives (SLOs) under 10 milliseconds end-to-end.

The central engineering challenge of a feature cache is maintaining consistency with the source of truth in the online store. Common strategies include:

• Time-to-Live (TTL): Entries expire after a fixed duration, accepting eventual consistency for staleness.
• Write-Through: The cache is updated synchronously when the online store is written, ensuring strong consistency at the cost of write latency.
• Event-Driven Invalidation: A Change Data Capture (CDC) stream from the source database triggers targeted cache entry purges or updates, balancing freshness with performance.

Feature vectors are stored and retrieved using a deterministic composite key that uniquely identifies the entity and the feature view. A typical key format is entity_type:entity_id:feature_view_version, such as user:48291:user_behavioral_profile_v3. This granular addressing allows the cache to invalidate specific slices of a feature vector without flushing the entire entity, which is essential when different feature groups have distinct update cadences and freshness requirements.

A small subset of entities—such as viral products or celebrity users—can generate a disproportionate volume of inference requests, creating hotspots that overwhelm specific cache shards. Mitigation techniques include:

• Replication: Duplicating the cached vector for a hot entity across multiple nodes.
• Local In-Memory Caching: Storing the hottest vectors directly in the model serving pod's process memory to eliminate network calls entirely.
• Request Coalescing: Merging concurrent requests for the same uncached key into a single backend query.

The choice of serialization protocol directly impacts cache memory density and deserialization CPU cost. Apache Arrow or FlatBuffers enable zero-copy deserialization, allowing the model to read feature values directly from the cache's byte buffer without parsing. This contrasts with text-based formats like JSON, which incur significant overhead. For numerical feature vectors, storing raw float32 arrays in a binary blob minimizes both memory footprint and access latency.

When a new model version is deployed, its feature cache is initially cold, causing a thundering herd of requests to the online store and elevated p99 latency. Cache warming pre-populates the cache by asynchronously fetching feature vectors for the most active entities from the offline store or a recent production traffic replay. This ensures the model serves with peak performance from its first production request, avoiding a degraded customer experience during rollout.

The low-latency database that physically persists pre-computed feature vectors for real-time serving. The feature cache sits in front of this store to absorb read traffic.

• Role: Primary persistence layer for hot features
• Examples: Redis, DynamoDB, Bigtable
• Latency: Typically single-digit milliseconds
• Trade-off: Higher latency than cache, but provides durability

The low-latency retrieval process that delivers feature vectors to a model endpoint during an online prediction request. The cache is a critical optimization layer in this path.

• Flow: Model → Serving API → Cache (L1) → Online Store (L2)
• Protocol: Typically gRPC for minimal overhead
• Goal: Retrieve hundreds of features in single-digit milliseconds
• Cache Hit: Returns features without touching the primary store

The process of pre-computing feature values from source data and persisting them into the online store. The cache must be invalidated or refreshed when new materialization completes.

• Trigger: Scheduled batch jobs or streaming writes
• Challenge: Ensuring cache consistency post-materialization
• Strategy: Time-to-live (TTL) alignment with feature freshness SLAs
• Impact: Stale cache entries cause training-serving skew

A metric defining the maximum acceptable age of a feature value in the online store. This directly governs cache TTL policies.

• Real-time features: Freshness < 1 second (cache TTL near zero)
• Near-real-time features: Freshness < 5 minutes (short TTL)
• Batch features: Freshness of hours or days (long TTL, high cache hit rate)
• Violation: Serving stale data degrades model accuracy

Feature values computed incrementally on real-time event streams using engines like Apache Flink or Kafka Streams. These features challenge cache design due to rapid mutation.

• Examples: Session click count, cart value, last-viewed category
• Cache Implication: Frequent invalidations reduce hit rate
• Pattern: Write-through cache updated simultaneously with online store
• Benefit: Captures immediate user intent for personalization

A one-dimensional array of numerical values representing all features for a specific entity at a point in time. The cache stores these vectors as serialized objects.

• Structure: [0.12, 1.0, 0.45, ...] with fixed schema
• Serialization: Protobuf or Avro for compact in-memory storage
• Size: Can range from hundreds to tens of thousands of dimensions
• Cache Key: Typically entity_id:feature_view_version