Model Serving

Online inference (real-time inference) is a synchronous serving pattern where predictions are generated and returned with low latency, typically under a few hundred milliseconds, in direct response to individual user requests via an API.

• Primary Use Case: User-facing applications requiring immediate feedback (e.g., fraud detection, product recommendations, chat responses).
• Key Challenge: Maintaining consistent latency SLAs under variable load, often requiring autoscaling and optimized runtimes.
• Infrastructure: Deployed as a scalable service (e.g., containerized microservice) behind a load balancer.

Batch inference is an asynchronous pattern where predictions are generated for large, accumulated datasets on a scheduled or triggered basis, prioritizing high throughput and cost-efficiency over low latency.

• Primary Use Case: Processing historical data, generating nightly reports, pre-computing embeddings for a search index, or scoring large customer segments.
• Execution: Jobs are typically orchestrated by workflow engines (e.g., Apache Airflow) and run on transient compute clusters (e.g., Spark, AWS Batch).
• Advantage: Enables efficient use of hardware by fully utilizing GPU/CPU capacity during processing.

Asynchronous inference is a pattern for long-running predictions where clients submit a request, receive a job ID, and poll for results later. This decouples request submission from result retrieval.

• Primary Use Case: Computationally expensive models (e.g., high-resolution image generation, complex simulations, large document summarization) where processing may take seconds or minutes.
• Workflow: 1) Client POSTs request to a job queue. 2) Server processes job. 3) Client GETs results using the job ID.
• Infrastructure: Requires a job queue (e.g., Redis, RabbitMQ), a result store, and worker processes.

Multi-adapter inference is a specialized serving architecture for Parameter-Efficient Fine-Tuning (PEFT) models. A single frozen base model (e.g., a large language model) can dynamically load different lightweight adapter modules (e.g., LoRA weights) per request.

• Primary Use Case: Efficiently serving hundreds of fine-tuned model variants for different tasks, customers, or domains from a shared base model.
• Key Mechanism: Runtime adapter injection, where the serving system swaps adapter weights into the model's computation graph on-the-fly.
• Benefit: Dramatically reduces storage and memory footprint compared to serving full copies of each fine-tuned model.

Shadow deployment is a safe validation pattern where a new model version processes live input traffic in parallel with the production model, but its predictions are logged and not returned to end-users.

• Primary Use Case: Performing live A/B testing for ML and drift detection without the risk of impacting user experience. It validates performance on real-world data before a canary release.
• Process: The serving infrastructure duplicates each incoming request, sending one copy to the production model and one to the shadow model. Only the production model's output is served.
• Outcome: Provides a realistic performance comparison and helps identify issues with data drift before customer exposure.

Edge inference is the deployment and execution of machine learning models directly on end-user devices (phones, IoT sensors) or local servers (retail stores, factories), rather than in a centralized cloud.

• Primary Use Case: Applications requiring ultra-low latency, offline functionality, bandwidth reduction, or enhanced data privacy (e.g., real-time object detection on smartphones, predictive maintenance on factory equipment).
• Technical Requirement: Models must be heavily optimized via on-device model compression techniques like quantization and pruning to run on constrained hardware.
• Challenge: Managing and updating a fleet of decentralized model versions without centralized control.

An inference endpoint is a hosted, network-accessible API (e.g., REST or gRPC) that receives input data and returns predictions from a deployed model. It is the primary interface for applications to consume model predictions.

• Key Protocols: REST/HTTP and gRPC are standard.
• Load Balancing: Endpoints are typically fronted by load balancers to distribute traffic across multiple model server instances.
• Security: Requires authentication (API keys, OAuth), encryption (HTTPS/TLS), and input validation to prevent adversarial attacks.

A model registry is a centralized repository for managing the lifecycle of ML models, including versioning, stage transitions (staging to production), and metadata. An artifact store is versioned storage (e.g., S3) for model binaries, datasets, and evaluation reports.

• Versioning: Tracks model iterations, code, data, and hyperparameters for full reproducibility.
• Lineage: Logs the data and pipeline run that produced each model version.
• Promotion Workflow: Enforces governance by requiring approval to move a model to a production stage.

The layer that manages the deployment, scaling, and lifecycle of model server instances, typically using Kubernetes and custom resource definitions (CRDs).

• Horizontal Pod Autoscaling (HPA): Dynamically adjusts the number of running pods based on CPU, memory, or custom metrics like request queue length.
• Resource Management: Ensures model servers have guaranteed GPU memory and compute allocations.
• Health Checks & Probes: Uses liveness and readiness probes to automatically restart unhealthy containers and manage traffic routing.

The telemetry system that collects, aggregates, and visualizes metrics and logs from the serving infrastructure and the models themselves.

• Key Metrics: Prediction latency (p50, p95, p99), throughput (requests/sec), error rates, and compute utilization (GPU/CPU).
• Drift Detection: Statistical monitoring for data drift and concept drift in live input data.
• Business Metrics: Integration with application logs to track the impact of predictions on core business KPIs.

A specialized serving architecture for Parameter-Efficient Fine-Tuning (PEFT) models, where a single base model can dynamically load different lightweight adapter modules (e.g., LoRA, Adapters) per request.

• Runtime Adapter Injection: Adapter weights are dynamically loaded and combined with the frozen base model at inference time, avoiding the need for a separate compiled model per task.
• Adapter Versioning: Tracks versions of adapter modules independently from the base model for deployment and rollback.
• Use Case: Enables efficient multi-tenant or multi-task serving from a shared base model, drastically reducing memory footprint compared to hosting multiple full models.

Online inference (or real-time inference) is a model serving pattern where predictions are generated synchronously and returned with low latency in response to individual user requests, typically via an API. This is essential for interactive applications like chatbots, fraud detection, and recommendation systems.

• Key Characteristic: Requires sub-second latency, often governed by a Latency SLA.
• Architecture: Typically involves a stateless, scalable endpoint that can handle variable request loads.
• Contrast with Batch: Differs from batch inference, which processes large datasets asynchronously for throughput over speed.

An inference endpoint is a hosted, network-accessible API (e.g., a REST or gRPC service) that receives input data and returns predictions from a deployed machine learning model. It is the primary interface between applications and a served model.

• Implementation: Can be a containerized service using frameworks like Triton Inference Server or vLLM.
• Management: Often registered and versioned within a Model Registry.
• Critical Metrics: Its performance is measured by latency, throughput (requests per second), and cost per inference.

A model registry is a centralized repository for managing the lifecycle of machine learning models. It provides governance and lineage tracking for model deployment.

• Core Functions: Model versioning, stage transitions (e.g., staging to production), metadata storage, and artifact linking.
• Integration Point: Works with an Artifact Store to track the exact model binary, dataset, and code used for training.
• For PEFT: Enables adapter versioning, tracking lightweight PEFT modules alongside their base model.

Dynamic batching is an inference optimization technique where a serving server groups multiple incoming requests into a single batch for parallel processing on hardware accelerators like GPUs. This maximizes hardware utilization and improves overall throughput.

• How it Works: The server waits for a short time window to collect requests, forms a batch, executes the model once, and then de-batches the results.
• Advanced Form: Continuous batching is a more advanced variant used for LLMs, where new requests are added to a running batch as prior requests finish generation steps.
• Benefit: Crucial for cost-effective serving, as it reduces the cost per inference by improving GPU efficiency.

Multi-adapter inference is a specialized serving architecture for Parameter-Efficient Fine-Tuning (PEFT) models. It allows a single base model to dynamically load and execute different lightweight adapter modules (e.g., LoRA weights) per request.

• Efficiency: Avoids the need to deploy a full, separate model for each fine-tuned task or tenant, saving significant memory and compute.
• Mechanism: Enabled by runtime adapter injection, where adapter weights are dynamically combined with the frozen base model's weights at inference time.
• Use Case: Ideal for multi-tenant SaaS platforms or applications requiring personalized models (e.g., different legal domains, customer styles).

Drift detection is the automated process of monitoring production model inputs and outputs to identify significant changes that degrade performance. It is a core component of production ML observability.

• Types of Drift:
  • Data Drift: Change in the statistical properties of input features.
  • Concept Drift: Change in the relationship between inputs and the target variable.
• Implementation: Uses statistical tests (e.g., Kolmogorov-Smirnov) or ML-based detectors on live traffic.
• Action: Triggers alerts in a monitoring dashboard and can initiate model retraining or rollback workflows.