Kubeflow

The Kubeflow Pipelines (KFP) component provides a platform for building and deploying portable, scalable ML workflows based on Docker containers. A pipeline is a description of an ML workflow, including all of the components and how they combine to form a directed acyclic graph (DAG).

• Python SDK: Define pipelines using the KFP domain-specific language, which compiles to Argo Workflows.
• Reusable Components: Package individual steps (data preprocessing, training, evaluation) as containerized components with defined inputs and outputs.
• Artifact Tracking: Automatically tracks lineage of datasets, models, and metrics between pipeline steps using ML Metadata.
• Caching: Enable step caching to skip re-execution of unchanged components, dramatically accelerating iterative development.

Kubeflow integrates JupyterHub to provide a multi-tenant environment for interactive data science. Each user receives a dedicated notebook server pod with isolated resources, enabling collaborative exploration without interference.

• Custom Images: Launch notebooks with pre-configured ML frameworks (TensorFlow, PyTorch) and custom libraries.
• GPU Allocation: Request specific GPU resources per notebook server through Kubernetes resource limits.
• Persistent Workspaces: Attach persistent volumes to preserve notebooks and data across server restarts.
• Identity-Aware Proxy: Securely expose notebook endpoints with authentication integrated via OIDC or Istio.

Kubeflow provides custom Kubernetes operators that manage distributed training jobs, handling pod scheduling, network configuration, and failure recovery automatically.

• TFJob: Orchestrates TensorFlow distributed training with parameter server and worker architectures.
• PyTorchJob: Manages PyTorch distributed training using the torch.distributed backend with master-worker topology.
• MPIJob: Supports Horovod and other MPI-based distributed training frameworks for all-reduce parallelism.
• Gang Scheduling: Ensures all workers in a distributed job are scheduled simultaneously to prevent resource deadlocks, using schedulers like Volcano.

Kubeflow supports multi-framework model serving with production-grade features like canary deployments, autoscaling, and explainability.

• KServe: A serverless inference platform that provides a standardized API for serving models from TensorFlow, PyTorch, ONNX, and scikit-learn.
• InferenceGraph: Define complex inference pipelines that chain multiple models together with conditional routing and ensembling logic.
• Transformer Pattern: Deploy pre-processing and post-processing containers alongside model servers to handle feature engineering and output formatting.
• Canary Rollouts: Gradually shift traffic between model versions to validate new deployments before full promotion.

Katib is a Kubernetes-native automated machine learning (AutoML) system that performs hyperparameter optimization and neural architecture search. It is framework-agnostic and supports any ML library.

• Search Algorithms: Choose from random search, grid search, Bayesian optimization, hyperband, and evolutionary algorithms.
• Early Stopping: Automatically terminate unpromising trials to conserve compute resources.
• Metrics Collection: Define objective metrics (accuracy, loss, latency) that Katib optimizes against, with a persistent UI for comparing trial results.
• Distributed Trials: Each hyperparameter combination runs as an independent Kubernetes job, enabling massive parallel exploration.

ML Metadata (MLMD) is the backbone service that records metadata and relationships between artifacts and executions across all Kubeflow components. It provides a complete audit trail for ML workflows.

• Artifact Lineage: Track exactly which dataset and hyperparameters produced a given model, enabling full reproducibility.
• Execution Graph: Visualize the DAG of all pipeline runs, including inputs, outputs, and intermediate artifacts.
• Cross-Component Integration: Pipelines, Katib, and KServe all write to the same metadata store, creating a unified view of the ML lifecycle.
• Query API: Programmatically retrieve metadata to build custom dashboards or trigger downstream automation based on artifact properties.

The core workflow engine of Kubeflow, enabling the definition, orchestration, and monitoring of Directed Acyclic Graphs (DAGs) for ML tasks. Each step in a pipeline runs in its own container, ensuring reproducibility and modularity.

• Uses Argo Workflows as the underlying orchestration engine
• Pipelines are defined as Python DSL code and compiled to YAML
• Supports artifact passing between steps via MinIO or S3-compatible storage
• Enables caching of step outputs to skip redundant executions
• Critical for automating training, batch inference, and data preprocessing

A serverless inference platform that provides a standardized API for deploying and serving ML models on Kubernetes. KFServing handles autoscaling, canary rollouts, and model versioning out of the box.

• Supports multiple frameworks: TensorFlow, PyTorch, ONNX, XGBoost
• Implements predictor, transformer, and explainer interfaces
• Uses Knative for scale-to-zero and request-based autoscaling
• Enables InferenceGraphs for complex multi-model serving topologies
• Critical for production-grade model serving in air-gapped environments