Artifact Store

Every artifact is stored with a unique, immutable identifier (e.g., a hash or sequential version). This creates a permanent, auditable record of every pipeline run's outputs, including:

• Model binaries (e.g., .pt, .onnx files)
• Processed datasets and data splits
• Evaluation reports and performance metrics
• Preprocessing artifacts like fitted scalers or tokenizers

This prevents accidental overwrites and is the foundation for model reproducibility and rollback capabilities.

Beyond the file itself, an artifact store captures rich metadata and provenance. This creates a graph linking artifacts to their origins, essential for debugging and compliance.

Key linked metadata includes:

• Code commit hash that generated the artifact
• Hyperparameters and configuration used for training
• Input dataset versions used
• Environment specifications (e.g., Docker image, library versions)
• Execution context (user, timestamp, pipeline run ID)

This turns the store into a searchable knowledge base for the model lifecycle.

The store abstracts underlying cloud object storage (e.g., Amazon S3, Google Cloud Storage, Azure Blob Storage) or a distributed file system (e.g., HDFS) into a single logical repository. This provides:

• Scalability to petabytes of model artifacts.
• Durability and high availability via cloud provider SLAs.
• Cost-effectiveness through object storage tiering.
• A standardized API (e.g., MLflow's log_artifact, Kubeflow's MinIO) for all pipeline components to interact with storage uniformly, decoupling logic from infrastructure.

A robust artifact store does not prescribe specific machine learning frameworks or file formats. It can version and manage outputs from any tool in the stack.

Common stored formats include:

• Serialized models: PyTorch .pt, TensorFlow SavedModel, Scikit-learn .pkl
• Open standards: ONNX, PMML
• Data files: Parquet, CSV, TFRecords
• Documents: JSON evaluation reports, HTML visualizations, PDFs

This flexibility future-proofs the MLOps platform as new frameworks and standards emerge.

The artifact store is not a standalone system; it is a core, integrated component of the ML pipeline orchestration engine. Orchestrators like Kubeflow Pipelines, Apache Airflow, or MLflow Projects automatically:

• Log outputs from each pipeline step as artifacts.
• Retrieve artifacts from previous steps as inputs for downstream tasks.
• Pass artifact references (URIs) through the workflow DAG.

This tight integration automates lineage creation and ensures the pipeline's data flow is explicitly captured and managed.

The versioned artifact serves as the single source of truth for deployment and monitoring systems. It acts as a controlled gateway between development and production.

Key integrations include:

• Model Registry: Promotes a specific artifact version from staging to production.
• Serving Infrastructure: Inference endpoints (e.g., Triton, vLLM) pull the approved model binary from the store.
• Monitoring Tools: Drift detection systems reference the training dataset artifact to compare against live data.
• CI/CD for ML: Automated pipelines trigger tests and deployments based on new artifact versions.

This central role enforces governance and prevents unvetted models from reaching production.

The primary artifact is the serialized model file itself. This includes:

• Full model weights (e.g., .pt, .h5, .safetensors files).
• Training checkpoints, allowing rollback to a previous state.
• Model metadata like the framework version, creation timestamp, and a unique hash (e.g., SHA-256).
• For PEFT, this includes the adapter weights (e.g., LoRA matrices) separately from the frozen base model, enabling modular deployment.

Stores quantitative and qualitative results to justify model promotion. Key artifacts include:

• Performance metrics (accuracy, F1, loss curves) logged as JSON/CSV.
• Confusion matrices and ROC curves for classification tasks.
• Model cards documenting intended use, limitations, and ethical considerations.
• A/B test results and shadow deployment logs comparing candidate models against the current champion.

Captures the exact data used for training and evaluation to ensure reproducibility.

• Feature-engineered datasets (e.g., parquet files, TFRecords).
• Train/Validation/Test splits with their identifiers.
• Data preprocessing artifacts, such as fitted Scalers, Encoders, and Vectorizers, saved as pickle files or ONNX.
• Data version links (e.g., a DVC pointer or a timestamp) to trace lineage back to raw sources.

The "recipe" for creating the model, stored to guarantee identical rebuilds.

• Pipeline definition files (e.g., Kubeflow YAML, Airflow DAGs).
• Hyperparameters used for the training run, stored as JSON/YAML.
• Environment specifications, such as conda-environment.yml or Docker image digests.
• Training scripts and notebooks versioned at the commit hash used for execution.

Optimized assets required for low-latency production serving.

• Compiled model formats like ONNX, TensorRT plans, or TorchScript.
• Quantized models (INT8/FP16) for efficient edge deployment.
• Inference server configuration (e.g., for Triton or vLLM).
• For PEFT, this includes the merged model or the adapter bundle for runtime adapter injection.

The connective tissue that links all other artifacts, creating an auditable trail.

• Experiment tracking metadata (link to MLflow Run ID, Weights & Biases run).
• Artifact provenance detailing which code, data, and parameters produced a given model binary.
• Deployment records linking a model version to a specific inference endpoint and canary release stage.
• Governance checks, such as bias assessment reports or compliance documentation.