Serving API

Serving APIs must deliver feature vectors in single-digit milliseconds to avoid becoming the bottleneck in real-time inference pipelines. This is typically achieved using gRPC rather than REST due to its use of HTTP/2 multiplexing and Protocol Buffers serialization, which significantly reduces payload size and connection overhead compared to JSON over HTTP/1.1.

• gRPC unary calls: Standard request-response for single entity lookups
• gRPC bidirectional streaming: For high-throughput scenarios requiring continuous feature delivery
• Connection pooling: Maintains persistent TCP connections to the online store to eliminate handshake latency
• Deadline propagation: Ensures that if a feature request exceeds its SLA, the entire prediction request fails fast rather than hanging

A single prediction often requires features for multiple entities—such as a user, a product, and a session context. The Serving API must support multi-key lookups in a single request to avoid the N+1 query problem that would otherwise multiply latency.

• Composite entity keys: Retrieve features for (user_id, product_id, context_id) simultaneously
• Vectorized response format: Returns a dense numerical array ready for direct model consumption without client-side transformation
• Feature selection: Allows the caller to specify a subset of features from a feature view, reducing payload size for models that only need specific inputs
• Partial failure handling: If one entity's features are unavailable, the API returns what it can with clear error metadata rather than failing the entire request

During online inference, the Serving API must guarantee that all feature values reflect the latest available state without temporal inconsistencies. For example, a user's session features and their historical aggregate features must be logically aligned to the same request timestamp.

• Timestamp-bound queries: Each request specifies a logical time, and the API retrieves the most recent feature value at or before that moment
• Atomic feature views: All features in a single response are guaranteed to be consistent as of the same materialization window
• Staleness metadata: The response includes a max_age_ms field indicating the freshness of the returned features, allowing the model to weigh predictions accordingly
• Read-your-writes: If a streaming feature was just updated by the current user's action, the API ensures that subsequent reads reflect that write

Not all features can be pre-materialized. The Serving API must support on-demand transformations where raw request-time context—such as the user's current geolocation or device type—is passed in the request payload and transformed into a feature vector by user-defined functions.

• Request-level context injection: Pass ephemeral data like user_agent or current_page_url directly in the API call
• Serverless UDF execution: The feature store executes transformation logic at request time without requiring a separate compute service
• Hybrid retrieval: Combines pre-computed features from the online store with on-demand computed features into a single unified vector
• Caching for on-demand features: Frequently computed on-demand features with stable inputs can be cached to reduce redundant computation

In multi-tenant or globally distributed deployments, feature data is often partitioned by entity or geography. The Serving API must route requests to the correct physical shard or region transparently, based on the entity key.

• Key-based sharding: Routes user_id: 12345 to the shard that owns that key range
• Region-aware endpoints: Directs requests to the nearest geographic replica to minimize network round-trip time
• Service mesh integration: Leverages sidecar proxies for automatic load balancing, circuit breaking, and retries without application code changes
• Hot-key mitigation: Detects and caches frequently accessed entity features to prevent single-shard overload

Every Serving API request must emit granular telemetry to enable latency debugging, feature drift detection, and usage auditing. This is critical for maintaining SLAs in production machine learning systems.

• Per-feature latency histograms: Measures retrieval time for each individual feature in a vector to identify slow components
• Feature access frequency: Tracks which features are actually consumed by models to identify unused or deprecated features
• Error rate by entity: Monitors failure rates segmented by entity key to detect data quality issues affecting specific user cohorts
• Distributed tracing: Propagates trace context through the full inference pipeline—from API gateway to feature store to model server—for end-to-end latency analysis
• Feature value distribution monitoring: Compares online feature distributions against training distributions to detect drift in real-time

The Serving API typically implements gRPC for its binary protocol and HTTP/2 multiplexing. Unary RPC sends a single request for a single feature vector, ideal for standard predictions. Server-side streaming RPC allows the online store to push a continuous flow of feature updates to a model, critical for streaming features where user intent changes mid-session. This avoids the overhead of constant polling and reduces tail latency.

A production Serving API enforces strict Service Level Agreements (SLAs). Typical guarantees include:

• p99 latency under 10ms for cached feature vectors
• Availability of 99.99% with circuit breakers to prevent cascading failures
• Deadlines propagated via gRPC metadata to cancel stale requests If the online store cannot meet the deadline, the API must fail fast rather than causing head-of-line blocking in the model server.

To minimize CPU overhead during deserialization, modern Serving APIs use Apache Arrow Flight or direct Arrow record batches. Unlike Protocol Buffers or JSON, Arrow is a columnar, in-memory format that eliminates serialization costs entirely. The feature vector is laid out in memory exactly as the model expects it, enabling zero-copy data transfer between the feature store and the inference runtime.

The Serving API must resolve a logical entity ID (e.g., user_id) to a physical storage partition. This involves:

• Consistent hashing to map entities to specific online store shards
• Affinity routing to direct requests to the nearest replica
• Composite key resolution for multi-entity feature views (e.g., user_id + product_id) This routing layer ensures that feature retrieval scales horizontally without hotspots.

Before returning a feature vector, the Serving API acts as a validation gate to enforce data quality. It checks:

• Feature freshness: Is the value older than the max age defined in the registry?
• Null handling: Should missing values return a default, an error, or be imputed?
• Schema enforcement: Does the returned vector match the expected feature view schema? This prevents garbage data from silently poisoning model predictions in production.

Not all features can be pre-materialized. The Serving API supports on-demand feature functions that compute values at request time using raw context passed in the API call. For example, a user's real-time GPS coordinates are transformed into a geohash or distance-to-store metric. This logic runs within the API's execution sandbox, ensuring the model receives a fully formed vector without coupling the inference code to raw data sources.