Seldon Core

Seldon Core enables the orchestration of sophisticated inference pipelines, known as Seldon Deployments. These are not single models but directed acyclic graphs (DAGs) where nodes can be:

• Models: Trained ML models (TensorFlow, PyTorch, SKLearn, etc.).
• Transformers: Pre- and post-processing components for feature engineering.
• Routers: Logic for A/B testing, multi-armed bandits, or ensemble methods.
• Combiners: Aggregators that merge outputs from multiple model branches. This allows engineers to build production-grade pipelines that mirror real-world business logic, such as chaining a fraud detection model with a recommendation model.

The platform provides enterprise-grade rollout patterns directly on Kubernetes, abstracting away complex YAML configurations. Key strategies include:

• Canary Rollouts: Safely deploy a new model version to a small percentage of traffic (e.g., 5%) to monitor performance before full promotion.
• Shadow Deployment: Send a copy of live traffic to a new model for comparison without affecting user-facing predictions.
• Multi-Armed Bandit Testing: Dynamically route traffic to the best-performing model variant based on real-time feedback, optimizing for business metrics. These strategies are managed through a custom Kubernetes SeldonDeployment resource, providing a declarative API for ML ops.

Seldon Core integrates directly with the Alibi and Alibi Detect open-source libraries to provide production-ready explainability and monitoring.

• Explainers: Generate feature attribution explanations (e.g., SHAP, Anchor, Integrated Gradients) for individual predictions to debug model behavior.
• Outlier Detectors: Identify anomalous inference requests that fall outside the model's trained data distribution using methods like Mahalanobis distance or Isolation Forests. These components can be added as sidecar containers to any model in the inference graph, providing actionable insights without modifying the core model code.

The platform automatically exposes a comprehensive set of metrics for monitoring model performance and infrastructure health.

• Request-Level Metrics: Latency, throughput, and 4xx/5xx error rates for each model in a graph.
• Custom Business Metrics: Users can emit arbitrary metrics (e.g., prediction value distributions) from within model code.
• Integration with Prometheus & Grafana: All metrics are exposed in Prometheus format, enabling seamless integration with standard Kubernetes monitoring stacks.
• Distributed Tracing: Supports OpenTelemetry for tracing requests as they flow through complex inference graphs, identifying latency bottlenecks.

Seldon Core serves models over multiple protocols to suit different client needs and optimize performance.

• REST/gRPC Endpoints: Automatic generation of both RESTful JSON APIs and high-performance gRPC endpoints from the same model packaging.
• gRPC for High Throughput: The gRPC interface, using protocol buffers, is significantly faster for high-volume inference, reducing serialization overhead.
• Seldon Prediction Protocol: A consistent internal wire format that standardizes payloads between components in an inference graph, regardless of the original model framework.

Seldon Core is designed as a set of Kubernetes operators and custom resource definitions (CRDs), making it a natural extension of the Kubernetes ecosystem.

• SeldonDeployment CRD: The primary resource for defining inference graphs, leveraging Kubernetes for lifecycle management.
• Operator Pattern: Controllers automatically reconcile the desired state, managing the creation of Kubernetes Deployments, Services, and Horizontal Pod Autoscalers.
• Integration with Service Meshes: Works seamlessly with Istio or Linkerd for advanced traffic management, security (mTLS), and observability features at the service mesh layer.

A directed acyclic graph (DAG) that defines a multi-step prediction pipeline. This is a core abstraction in Seldon Core. A graph can include:

• Pre-processing transformers
• One or more model predictors
• Post-processing transformers
• Routers for A/B tests
• Combiners to merge outputs This allows you to build complex, modular inference workflows (e.g., a recommender that first filters, then ranks, then explains) as a single deployable Kubernetes resource.

A release strategy where a new version of a model is deployed to a small, controlled subset of live traffic (e.g., 5%). Seldon Core implements this natively using its traffic splitting capabilities. You can:

• Specify the percentage of requests routed to the new (canary) model.
• Monitor its performance and business metrics.
• Gradually increase traffic or roll back based on results. This minimizes risk by validating new models in production before a full rollout.

A system component responsible for managing the lifecycle, scheduling, and coordination of multiple model serving instances. Seldon Core acts as a model orchestrator on Kubernetes. Its responsibilities include:

• Translating high-level deployment specs into Kubernetes objects.
• Managing rolling updates and rollbacks.
• Scaling replicas up/down based on load.
• Ensuring health and readiness of model servers. It abstracts away the complexity of managing individual pods and services.