Model Versioning

The core serialized file containing the trained parameters (weights and biases) of the neural network. This is the output of the training process. Common formats include:

• PyTorch .pt or .pth: Contains the model's state_dict.
• TensorFlow SavedModel: A directory with the model's computation graph and variables.
• ONNX .onnx: A standardized, framework-agnostic format for model interchange.
• TensorRT Plan .engine: A highly optimized execution plan for NVIDIA GPUs. The artifact is the primary payload for the inference server.

The execution logic that loads the artifact and performs the forward pass. This includes:

• Preprocessing/Postprocessing Scripts: Code to transform raw input into model-ready tensors and convert outputs into a usable format.
• Framework Runtime: The specific library versions (e.g., PyTorch 2.1.0, TensorFlow 2.15.0) required for compatibility.
• Dependencies: Other Python packages or system libraries the code depends on. This is often packaged as a Docker container to ensure the execution environment is identical across development, staging, and production.

A unique, immutable label assigned to the specific combination of artifact and code. This is the primary key for tracking and retrieval. Common schemes include:

• Semantic Versioning (e.g., fraud-detector-v2.1.3): Uses MAJOR.MINOR.PATCH to signal breaking changes, new features, and bug fixes.
• Commit Hash (e.g., model-abc123f): Ties the version directly to a Git commit for full traceability.
• Timestamp (e.g., 2024-11-05-14-30-00): Provides a chronological ordering. The identifier is used in API endpoints (e.g., /predict/v2.1.3) and for rollback procedures.

Structured data describing the model's characteristics and provenance. Essential metadata includes:

• Training Dataset: Identifier or fingerprint (e.g., dataset hash) of the data used for training.
• Performance Metrics: Validation accuracy, F1 score, latency benchmarks recorded during evaluation.
• Hyperparameters: Learning rate, batch size, optimizer settings used during training.
• Input/Output Schema: Expected data types, shapes, and ranges for the model's API.
• Author and Timestamp: Who created the version and when. This metadata is typically stored in a model registry (like MLflow or a custom database) and is critical for auditability and compliance.

Deployment-specific settings that dictate how the inference server should instantiate and run the model. This acts as the serving blueprint. Key configurations include:

• Resource Requests/ Limits: CPU, memory (RAM), and GPU requirements for Kubernetes.
• Batching Parameters: Maximum batch size, timeout windows, and padding preferences.
• Autoscaling Rules: Metrics and thresholds (e.g., requests per second > 100) for scaling the service.
• Health Check Endpoints: Paths for liveness and readiness probes.
• Logging and Monitoring: Settings for metrics collection (e.g., Prometheus) and structured logs. Tools like KServe's InferenceService YAML or Seldon Core's SeldonDeployment encapsulate this manifest.

Documented evidence of the model version's performance and safety before promotion to production. This is the gatekeeping artifact. Reports include:

• A/B Test Results: Comparison of key business metrics (e.g., conversion rate) against a previous champion model on a traffic slice.
• Bias/Fairness Audits: Analysis of performance disparities across protected subgroups (e.g., gender, ethnicity).
• Adversarial Robustness Tests: Performance under perturbed or maliciously crafted inputs.
• Integration Test Logs: Verification that the model works correctly with upstream data pipelines and downstream applications. These reports provide the quantitative justification for a deployment decision and are essential for MLOps governance.

Canary deployment is a release strategy where a new version of a model is initially deployed to a small, controlled subset of production traffic. This allows for validation of performance and stability before a full rollout.

• Mitigates risk by limiting the impact of a faulty model version.
• Enables A/B testing to compare new and old versions on live data.
• Uses traffic routing rules (e.g., 5% to v2, 95% to v1) based on user attributes or request load.
• Requires robust monitoring to detect regressions in accuracy or latency.

Online inference (or real-time inference) is a serving pattern where model predictions are generated synchronously and returned with low latency in response to individual, live user requests. This is the primary context for versioned models.

• Typical latency requirements range from milliseconds to a few seconds.
• Requires models to be pre-loaded and cached in memory (avoiding cold starts).
• Served via REST or gRPC API endpoints.
• Contrasts with batch inference, which processes large datasets asynchronously for high throughput.

Model monitoring is the continuous observation of a deployed model's performance, behavior, and operational health in production. For versioned models, it's essential for comparing iterations.

• Tracks key metrics: prediction accuracy, latency, throughput, and error rates.
• Detects concept drift (changing relationships between input and output) and data drift (changing input data distributions).
• Provides alerts when a new model version degrades compared to a baseline.
• Tools include Prometheus for metrics and specialized MLOps platforms.

Multi-tenancy in model serving is an architectural pattern where a single inference server or cluster simultaneously hosts and isolates multiple distinct models or model versions for different clients or use cases.

• Optimizes GPU and memory utilization by sharing resources.
• Requires isolation to prevent one model from impacting another's performance or security.
• Managed through resource quotas, separate execution environments, and namespacing.
• Critical for cost-effective serving of many versioned models at scale.

An API gateway is a reverse proxy that acts as a single entry point for client requests, routing them to appropriate backend model inference services. It is a key control point for managing versioned model endpoints.

• Routes requests (e.g., /predict/v1/ vs. /predict/v2/) to the correct model version.
• Handles cross-cutting concerns: authentication, rate limiting, request/response transformation, and logging.
• Enables canary deployments and A/B testing by applying traffic-splitting rules.
• Examples include Kong, Apache APISIX, and cloud-native offerings.