Workflow DAG (Directed Acyclic Graph)

The directed edges (arrows) between nodes explicitly define the execution order and data flow. A task (node) cannot execute until all of its upstream dependencies have completed successfully. This prevents race conditions and ensures data is available when needed.

• Example: A 'Feature Engineering' node must complete before a 'Model Training' node can begin, as the trainer requires the processed features as input.
• Key Property: This creates a partial ordering of tasks, which is essential for parallel execution where possible.

The graph is acyclic, meaning there are no circular paths or loops. You cannot start at one node, follow the directed edges, and return to the same node. This property guarantees that the pipeline will eventually terminate and prevents infinite execution cycles.

• Why it matters: Loops can cause deadlocks or uncontrolled resource consumption. The DAG's acyclic nature makes the workflow schedulable and its runtime predictable.
• Enforcement: Pipeline orchestration tools (e.g., Apache Airflow, Kubeflow Pipelines) validate DAGs to ensure they contain no cycles before allowing execution.

Each node in the DAG represents a single, atomic unit of work or computation. This is typically a containerized process, a Python function, or a SQL query. Tasks should be idempotent (producing the same result from the same inputs) to support retries and caching.

• Common ML Tasks: Data validation, feature extraction, model training, hyperparameter tuning, model evaluation, and deployment.
• Isolation: Failures are contained within a node, allowing the orchestrator to retry the specific task without restarting the entire pipeline.

A DAG's structure exposes opportunities for parallel execution. Nodes that do not depend on each other can run concurrently on available compute resources, dramatically reducing total pipeline runtime.

• Example: After a 'Data Fetch' node, separate 'Clean Dataset A' and 'Clean Dataset B' nodes with no dependency between them can run in parallel.
• Orchestrator Role: Tools like Apache Airflow analyze the DAG to identify and schedule independent tasks simultaneously, optimizing resource utilization.

The edges in a workflow DAG often represent the flow of data artifacts (e.g., serialized datasets, model files, evaluation metrics) from one task to its downstream consumers. This creates a clear, auditable data lineage.

• Implementation: Artifacts are typically stored in a shared, versioned artifact store (e.g., S3, Google Cloud Storage) with paths or metadata passed between nodes.
• Benefit: Any model prediction or metric can be traced back through the exact data and code that produced it, which is critical for debugging, compliance, and reproducibility.

The DAG itself is a declarative blueprint of the pipeline, defined as code (e.g., in Python). This code is version-controlled, enabling:

• Reproducibility: Any historical pipeline run can be recreated exactly from the committed DAG code and versioned artifacts.
• Collaboration: Teams can review, modify, and test pipeline changes via pull requests.
• CI/CD Integration: The DAG definition can be part of a CI/CD for ML process, where changes are automatically tested and deployed to a scheduler like Apache Airflow.

This foundational DAG orchestrates the core development lifecycle.

• Nodes include data validation, feature engineering, model training, hyperparameter tuning, and evaluation.
• Edges enforce order: data must be validated before feature creation; a model must be trained before it can be evaluated.
• A key dependency is the holdout validation set, which is split from the main data at the start and only flows into the evaluation node, preventing data leakage.
• Tools like Apache Airflow, Kubeflow Pipelines, and Metaflow are commonly used to define and execute these graphs.

This automated DAG triggers periodic model updates in production.

• It begins with a schedule or data arrival trigger.
• Key stages include fetching new training data, computing drift metrics against a reference dataset, and conditionally executing a retraining branch if drift exceeds a threshold.
• The DAG incorporates model validation and A/B testing against the current champion model before a canary deployment.
• This pattern is essential for maintaining model performance amidst concept drift and data drift without manual intervention.

This DAG structures complex, real-time prediction workflows often seen in ranking and recommendation systems.

• A candidate retrieval node (e.g., using a lightweight model or heuristic) filters thousands of items down to hundreds.
• A heavy ranking node (a larger, more accurate model) then scores the retrieved candidates.
• A final business logic node applies rules, diversifies results, or formats the output.
• Edges represent the flow of candidate lists between stages. This design optimizes cost and latency by applying compute-intensive models only to a pre-filtered subset.

This DAG handles high-volume, asynchronous prediction jobs on large datasets.

• It is triggered on a schedule (e.g., nightly) or by the arrival of a new data file.
• Core nodes are data partitioning, distributed model scoring across a compute cluster, and aggregation of results.
• Post-processing nodes are critical: they might join predictions with customer databases, apply business rules, or format outputs for downstream systems like data warehouses or email campaigns.
• This pattern prioritizes throughput and cost-efficiency over the low latency required for online inference.

This specialized DAG manages the lifecycle of Parameter-Efficient Fine-Tuned models, a key concern in the PEFT Deployment and MLOps content group.

• It starts with a base model being loaded from a registry.
• A core node performs runtime adapter injection, dynamically loading a specific LoRA or Adapter module's weights and merging them with the frozen base model.
• Subsequent nodes handle model compilation for the target hardware (e.g., GPU with TensorRT) and canary deployment to a multi-adapter inference server like vLLM or Triton.
• The DAG ensures efficient, multi-tenant serving where a single base model instance can serve numerous fine-tuned variants.

This comprehensive DAG integrates the entire ML lifecycle from experimentation to production monitoring.

• It spans data ingestion, feature store updates, model training/retraining, and model deployment.
• Crucially, it includes post-deployment nodes for continuous monitoring, logging predictions, and calculating performance metrics and drift detection scores.
• Conditional edges can trigger alerts or kick off a retraining pipeline if monitoring thresholds are breached.
• This DAG embodies CI/CD for ML, ensuring automated, auditable, and reliable model operations.

The automated coordination and execution of a sequence of interdependent tasks in a machine learning workflow. A Workflow DAG is the primary structure used by orchestration engines.

• Core Function: Manages dependencies, schedules tasks, handles retries, and monitors execution.
• Key Tools: Apache Airflow, Kubeflow Pipelines, and Metaflow are popular frameworks that use DAGs to define workflows.
• PEFT Context: Orchestrates the entire lifecycle of a PEFT model, from data preparation and adapter training to evaluation and deployment.

The automated practice of reliably and safely deploying new versions of machine learning models and their associated pipelines into production.

• Automation Goal: Extends CI/CD principles to ML, automating testing, packaging, and deployment of model artifacts and code.
• DAG Integration: The deployment pipeline itself is often modeled as a DAG, ensuring steps like integration testing, canary releases, and registry updates occur in the correct order.
• Critical for PEFT: Enables rapid, safe iteration of new adapter versions or base model updates.

A centralized repository for managing the lifecycle of machine learning models, including versioning, lineage tracking, stage transitions, and metadata storage.

• Single Source of Truth: Stores model binaries, code snapshots, datasets, and performance metrics.
• DAG Artifact: Trained models (and PEFT adapters) are the final artifacts produced by a training pipeline DAG and are stored in the registry.
• PEFT Specificity: Must track adapter versioning, linking adapter files to their specific base model version and training data.

A versioned storage system for machine learning pipeline outputs such as trained model binaries, datasets, evaluation reports, and preprocessing artifacts.

• Data Layer for DAGs: Each node (task) in a Workflow DAG reads from and writes artifacts to this store.
• Ensures Reproducibility: By storing immutable, versioned outputs of every pipeline step.
• PEFT Storage: Holds the serialized weights for base models and the much smaller adapter files (e.g., LoRA matrices), enabling efficient storage of many fine-tuned variants.

A serving architecture where a single base model can dynamically load and execute different lightweight PEFT adapter modules per request, enabling efficient multi-task or multi-tenant serving.

• Core Efficiency: Shares the memory footprint of the large base model across many tasks, only swapping in small adapter weights.
• DAG Link: The deployment of this architecture is managed by an MLOps pipeline DAG.
• Key Technique: Relies on runtime adapter injection to merge adapter weights with the base model at inference time without recompilation.

Continuous Integration and Continuous Delivery/Deployment for Machine Learning. Automates the testing, building, and deployment of ML models, data, and pipelines.

• Pipeline as Code: The Workflow DAG definition is part of the codebase, triggered by commits or data changes.
• Automated Gates: Includes steps like data validation, model unit tests, integration tests, and performance benchmarking within the pipeline.
• Foundation for PEFT MLOps: Ensures that new adapter training runs are automatically validated and ready for safe deployment.