Model Serving

The central catalog for trained model artifacts. It provides versioning, lineage tracking, and metadata storage (e.g., training data hash, metrics, framework). Key functions include:

• Artifact Storage: Stores serialized model files (e.g., .pt, .pb, .onnx).
• Version Control: Assigns unique identifiers (e.g., v1.2.3) to each model iteration.
• Stage Promotion: Manages lifecycle stages like Staging, Production, and Archived.
• Access Control: Governs who can deploy or modify models. Examples: MLflow Model Registry, Kubeflow Pipelines, Neptune.

The runtime service that loads a model and executes predictions. It exposes a network API (typically REST/gRPC) and handles request batching, hardware acceleration, and model lifecycle on the server.

• API Gateway: Provides a uniform endpoint (e.g., /predict) for clients.
• Model Loading: Dynamically loads and unloads models from the registry into memory.
• Hardware Optimization: Utilizes GPUs/TPUs and supports formats like TensorRT or OpenVINO.
• Multi-Model Support: Can host multiple models or versions concurrently on a single server. Common servers: TensorFlow Serving, TorchServe, Triton Inference Server.

The automation layer that manages the deployment and scaling of inference servers across a cluster. It translates high-level deployment specs (canary, A/B) into containerized workloads.

• Specification: Defines resources (CPU/GPU), replicas, and update strategies.
• Scheduling: Places model servers on available cluster nodes (e.g., using Kubernetes).
• Rollout Management: Executes gradual rollouts, canary releases, and automatic rollbacks based on health checks.
• Scaling: Triggers horizontal pod autoscaling based on request load or custom metrics.

The component that directs incoming prediction requests to the appropriate model version or endpoint. It is critical for implementing safe deployment patterns.

• Request Routing: Uses HTTP headers, cookies, or percentages to split traffic between model variants (e.g., 95% to v1, 5% to v2).
• Load Distribution: Evenly distributes requests across multiple replicas of the same model for high availability.
• Shadow Routing: Can duplicate (mirror) traffic to a challenger model for offline analysis without affecting the user response.
• Fallback Routing: Redirects requests to a stable fallback model if the primary endpoint fails health checks.

The collection of tools that provide real-time visibility into the serving system's health, performance, and prediction quality.

• Performance Metrics: Tracks latency (p50, p99), throughput (RPS), and error rates.
• Resource Metrics: Monitors GPU/CPU utilization, memory usage, and I/O.
• Business & ML Metrics: Logs model-specific metrics like prediction distributions, input drift, and custom evaluation scores.
• Logging & Tracing: Captures request/response payloads (often sampled) and provides distributed traces for debugging.
• Alerting: Triggers alerts on SLO violations (e.g., latency > 100ms) or drift detection.

A service that provides consistent, low-latency access to pre-computed features for model inference. It ensures the features served online match those used during model training.

• Online Serving: Delivers feature vectors via a high-speed API (e.g., Redis, DynamoDB).
• Point-in-Time Correctness: Retrieves feature values as they existed at the time of the prediction event.
• On-the-Fly Transformation: Applies necessary transformations (normalization, encoding) to raw request data before sending it to the model.
• Backfill Support: Enables consistent feature generation for training and inference pipelines.

A canary release is a deployment strategy where a new model version is initially rolled out to a small, specific subset of users or infrastructure to validate its performance and stability before a full rollout.

• Risk Mitigation: By limiting initial exposure, potential failures impact only a small fraction of traffic, containing blast radius.
• Validation: The canary's performance (latency, error rate, business metrics) is closely monitored and compared against the baseline. If metrics degrade, traffic is instantly rerouted back, and the rollout is halted.
• Implementation: Typically managed via traffic splitting rules in a service mesh (e.g., Istio) or API gateway, which routes a defined percentage of requests to the new endpoint.

Shadow mode (or dark launch) is a deployment technique where a new model processes live production traffic in parallel with the current model, but its predictions are logged and not used to affect user-facing decisions.

• Zero-Risk Validation: Allows for comparing the new model's predictions and performance (e.g., computational latency, output distribution) against the champion model under real-world load without any user impact.
• Data Collection: Crucial for gathering a labeled dataset for evaluation if the new model's predictions would have been correct, enabling offline A/B testing.
• Architecture: Implemented via traffic mirroring, where requests are duplicated and sent asynchronously to the shadow model endpoint.

A circuit breaker is a resilience pattern that protects a system from cascading failures. For model serving, it temporarily stops sending requests to a failing inference endpoint, allowing it time to recover.

• Mechanism: The breaker monitors for failures (e.g., timeouts, 5xx errors). After a failure threshold is breached, it "trips" and fails fast, returning a fallback response immediately without calling the downstream service.
• States: It cycles through Closed (normal operation), Open (requests fail immediately), and Half-Open (allowing a test request to see if the service has recovered).
• Integration: Often implemented in API gateways, service meshes, or client-side libraries like resilience4j or Hystrix to guard against unhealthy model servers.