Model Versioning

The unique identifier that distinguishes one model iteration from another. This is the primary key for tracking, auditing, and deploying a specific model snapshot.

• Semantic Versioning (e.g., v2.1.0): Uses MAJOR.MINOR.PATCH to signal breaking changes, new features, and bug fixes.
• Commit Hash (e.g., a1b2c3d): Ties the model directly to a specific code commit in a Git repository for precise lineage.
• Timestamp (e.g., 2024-11-05T14:30:00Z): Provides a chronological ordering for versions generated by automated pipelines.

The serialized file containing the trained model's learned parameters (weights and biases) and, optionally, its architecture. This is the core deployable object.

• Common Formats: SavedModel (TensorFlow), .pt or .pth (PyTorch), ONNX, or PMML.
• Contents: Includes the computational graph and the exact state of all parameters at the end of training.
• Storage: Typically stored in a model registry or object store (e.g., S3, GCS) with the version ID as part of the object key.

The exact source code used to train the model, frozen at the point of version creation. This is critical for reproducibility and debugging.

• Dependency Locking: Includes a requirements.txt, Pipfile.lock, or container image SHA to pin library versions (e.g., torch==2.1.0).
• Git Commit: The model version should be explicitly linked to the Git commit hash of the training script and any feature engineering code.
• Configuration Files: Hyperparameters, model architecture definitions, and data preprocessing steps are stored as code (e.g., YAML, JSON).

A reference to the specific dataset used for training and evaluation. This establishes the data lineage and is essential for auditing bias or performance regressions.

• Data Versioning: Uses a data versioning tool (e.g., DVC, LakeFS) or a snapshot ID from a feature store.
• Metadata: Records dataset statistics, schema, and any applied transformations or filters.
• Evaluation Split: The specific holdout set used for the reported performance metrics must be versioned alongside the training data.

The quantitative and qualitative evaluation results for this specific version, defining its expected behavior and establishing a baseline for monitoring.

• Evaluation Metrics: Key performance indicators (KPIs) like accuracy, F1-score, AUC, or business metrics recorded on the versioned test set.
• Bias & Fairness Reports: Results of fairness audits across protected attributes.
• Inference Characteristics: Baseline latency, throughput, and memory footprint measured in a standardized environment.

The infrastructure and runtime specifications required to serve the model version reliably in production. This decouples the model from its serving environment.

• Inference Environment: Specifies the container image, Python version, and framework dependencies for the inference endpoint.
• Resource Requirements: Defines CPU/memory/GPU needs and autoscaling policies.
• Serving Configuration: Includes any pre/post-processing logic, circuit breaker settings, and health check endpoints specific to this version.

Model versioning creates an immutable, timestamped record of every model artifact, enabling exact reproduction of past predictions and full audit trails. This is critical for debugging, regulatory compliance, and scientific rigor.

• Artifact Snapshot: Each version captures the exact model code, trained weights, hyperparameters, and the specific training data snapshot used.
• Lineage Tracking: It answers the question: "Which data and code produced this specific model prediction from six months ago?"
• Compliance: Essential for regulated industries (finance, healthcare) where model decisions must be explainable and reproducible for auditors.

Versioning is the mechanism that enables safe deployment strategies like canary releases and A/B testing by allowing traffic to be precisely routed between different, immutable model versions.

• Traffic Splitting: A load balancer can direct 5% of traffic to model:v2.1 while 95% stays on model:v2.0.
• Instant Rollback: If v2.1 shows a regression, operators can instantly revert 100% of traffic back to the known-good v2.0 by updating a single version pointer.
• Zero-Downtime Updates: Blue-green deployments switch traffic between two identical environments hosting different model versions.

Teams can develop, train, and evaluate multiple model candidates (e.g., different architectures, updated training data) concurrently without interfering with the stable production version.

• Champion-Challenger: The production champion (model:prod-v1) runs live while multiple challenger versions (model:exp-7, model:exp-8) are evaluated offline or in shadow mode.
• Isolation: Each experiment is a separate, versioned branch of the model lineage, preventing configuration conflicts.
• Informed Promotion: The best-performing challenger version is then promoted through staged deployment (Dev -> Staging -> Canary -> Prod).

By associating inference logs and performance metrics with a specific model version, teams can accurately detect regression and data drift.

• Version-Aware Metrics: Dashboards track key performance indicators (accuracy, latency, error rate) per model version, not just an aggregate endpoint.
• Pinpointing Regressions: A drop in performance is immediately correlated to the deployment of a specific version (v3.2), speeding up root cause analysis.
• A/B Test Analysis: Metrics are compared between version A and version B to statistically determine a winner.

Versioning is a core requirement of enterprise AI governance frameworks, providing the control plane for model lifecycle management and risk mitigation.

• Approval Workflows: New model versions (v4.0-rc1) can be gated, requiring approval from data science, legal, and business teams before promotion to production.
• Immutable Audit Log: The version registry logs who approved what version and when, creating a chain of custody.
• Regulatory Reporting: For laws like the EU AI Act, companies must demonstrate control over which models are deployed and provide documentation for each version.

A versioned model registry acts as a source of truth for recovery, ensuring a known-stable version can be rapidly redeployed in case of a catastrophic failure.

• Fallback Version Designation: A fallback model (e.g., a simpler, robust model:v1.4) is pre-designated in system configuration.
• Automated Rollback Triggers: Monitoring systems can be configured to automatically trigger a rollback to the fallback version if the primary version's error rate exceeds an SLO threshold.
• Artifact Reproducibility: In a total system rebuild, the exact production model can be reinstantiated from its version identifier in the registry.

A/B testing is a controlled experiment methodology for comparing two or more model versions (A and B) by randomly splitting live user traffic between them. The goal is to statistically determine which variant performs better against a predefined business or quality metric, such as click-through rate, conversion, or accuracy. It is a cornerstone of data-driven deployment decisions.

• Randomized Assignment: Users are randomly assigned to a test group to avoid selection bias.
• Primary Metric: A single, well-defined objective metric is chosen for evaluation.
• Statistical Significance: Tests run until results reach a confidence threshold (e.g., 95%). This method moves deployment decisions from intuition to empirical validation.

A canary release is a risk-mitigation deployment strategy where a new model version is initially rolled out to a small, non-critical subset of users or infrastructure—the 'canary.' Performance and system health are closely monitored using Service Level Objectives (SLOs). If the canary performs well, the rollout gradually expands; if issues are detected, it is immediately rolled back with minimal impact. This strategy is named after the historical use of canaries in coal mines to detect gas.

• Phased Rollout: Traffic percentage increases incrementally (e.g., 1% → 5% → 25% → 100%).
• Real-time Monitoring: Relies on metrics for latency, error rates, and business KPIs.
• Fast Rollback: Infrastructure allows instant reversion to the previous stable version.

Shadow mode (or dark launch) is a deployment technique where a new model processes a copy of live production traffic in parallel with the currently serving model. Its predictions are logged and analyzed but are not used to affect any user-facing decisions or actions. This allows for safe, zero-risk performance comparison and validation under real-world load and data conditions.

• Zero User Impact: The shadow model's outputs are purely observational.
• Realistic Validation: Evaluates performance on actual production data distribution.
• Infrastructure Stress Test: Can reveal scaling or latency issues before the model carries any load. It is often a precursor to a canary release or A/B test.

Traffic splitting is the technical mechanism for routing a controlled percentage of incoming inference requests to different model versions or endpoints. It is the infrastructure layer that enables A/B testing and canary releases. Splitting is typically managed by a load balancer, service mesh (like Istio), or feature flag system.

• Rule-Based Routing: Traffic can be split by user ID, geography, device type, or random hash.
• Dynamic Configuration: Split percentages can be changed in real-time without redeployment.
• Observability: Each route must be instrumented to collect separate performance metrics. This provides the granular control needed for gradual, safe rollouts.

A rollback strategy is a predefined, often automated, plan for reverting a system to a previous known-good state—such as a prior model version—in response to detected failures, performance regressions, or other critical issues. It is a fundamental component of resilient deployment. An effective strategy includes:

• Automated Triggers: Conditions that initiate a rollback (e.g., error rate > 5%, latency SLO breach).
• Pre-Validated Artifacts: The previous model version is already staged and tested.
• Immutable Versioning: The system can reliably redeploy a specific, immutable artifact.
• State Management: Consideration for any data or state compatibility issues during the reversion. Without a rollback strategy, mitigating a bad deployment becomes a high-stress, manual emergency.