Airflow DAG

The fundamental data structure of an Airflow workflow. A Directed Acyclic Graph (DAG) ensures tasks are executed in a specific order without cycles, preventing infinite loops.

• Nodes represent individual tasks (e.g., PythonOperator, BashOperator).
• Directed Edges define dependencies between tasks using >> and << operators.
• Acyclic property guarantees a finite, predictable execution path, which is critical for scheduling and debugging.

DAGs are defined declaratively as Python code. This Workflow-as-Code approach integrates orchestration into the software development lifecycle.

• The DAG object (DAG() or @dag decorator) acts as a container for tasks and global settings.
• Developers use Python's full expressiveness for dynamic workflow generation (e.g., creating tasks in loops).
• Code is version-controlled, peer-reviewed, and tested like any other application logic, ensuring reliability and auditability.

Explicit dependencies control the execution plan. Airflow's scheduler uses these to determine task readiness.

• Use task1 >> task2 to set task1 as upstream of task2.
• Complex patterns like parallel execution (fan-out) and conditional branching (using BranchPythonOperator) are supported.
• The scheduler respects these dependencies to build a topological sort, ensuring tasks run only when their upstream dependencies have succeeded.

DAGs are activated by schedules or external events, decoupling workflow definition from execution timing.

• Temporal Scheduling: Defined via schedule_interval using cron expressions or timedelta objects.
• Event-Driven Triggers: DAGs can be triggered via the Airflow REST API, CLI, or sensors that poll for external conditions.
• Manual Execution: DAG Runs can be initiated on-demand through the Airflow UI for testing or ad-hoc operations.

Operators define the work performed by a task. They abstract execution details, making DAGs modular and extensible.

• Built-in Operators: PythonOperator (executes Python callables), BashOperator (runs shell commands), SimpleHttpOperator (makes HTTP requests).
• Provider Packages: Extend Airflow with operators for AWS, GCP, Snowflake, Databricks, and hundreds of other services.
• Custom Operators: Engineers can create their own by subclassing BaseOperator to encapsulate proprietary logic.

Airflow provides comprehensive observability into DAG and task state, which is essential for production orchestration.

• Task States: Each task transitions through states like queued, running, success, failed, skipped.
• Centralized UI: The Airflow Webserver offers tree, graph, and timeline views of DAG runs, plus logs and task details.
• Retry Logic & Alerting: Tasks can be configured with automatic retry logic (exponential backoff) and failure notifications via email or Slack.

A Directed Acyclic Graph (DAG) is a finite directed graph with no cycles, used in workflow orchestration to model tasks as nodes and their dependencies as edges, ensuring a non-circular execution order. This mathematical structure is the foundational model for an Airflow DAG, providing the guarantee that workflows will terminate.

• Nodes represent individual tasks or operations.
• Directed Edges define dependencies and execution order.
• Acyclic Property prevents infinite loops, ensuring every workflow has a start and end.

A task orchestrator is a system component responsible for coordinating the execution, scheduling, and dependency management of individual tasks within a larger, automated workflow. Apache Airflow is a prime example of a task orchestrator that uses DAGs as its core abstraction.

• Key Functions: Scheduling, dependency resolution, execution, monitoring, and retry management.
• Contrast with Scheduler: An orchestrator manages the full lifecycle, while a scheduler primarily handles timing.
• Use Case: Essential for complex data pipelines, ETL processes, and multi-step machine learning workflows.

Workflow-as-Code is a development practice where workflow definitions are authored, versioned, and managed as code (e.g., in Python, YAML) within a standard software development lifecycle. Airflow DAGs are defined as Python scripts, epitomizing this practice.

• Benefits: Enables code review, CI/CD integration, unit testing, and easy refactoring.
• Contrast with GUI-based Designers: Provides greater flexibility, reproducibility, and integration with developer toolchains.
• Example: An Airflow DAG file (my_pipeline.py) checked into a Git repository.

Declarative orchestration is an approach where a workflow is defined by specifying the desired end state and dependencies, leaving the engine to determine the optimal execution sequence. This contrasts with imperative, step-by-step instructions. Airflow DAGs use a mix, declaring task dependencies but often using imperative code within tasks.

• Core Principle: Define what should happen, not how to make it happen step-by-step.
• Engine Responsibility: The orchestrator (like Airflow) resolves the execution graph from the declared dependencies.
• Related Pattern: Used in Kubernetes manifests and infrastructure-as-code tools like Terraform.

An execution plan is a runtime blueprint, generated from a workflow definition, that specifies the precise order, conditions, and resource assignments for carrying out a sequence of tasks. In Airflow, the scheduler parses a DAG to create an execution plan before queuing tasks for the workers.

• Runtime Artifact: Created by the scheduler from the static DAG definition.
• Contains: Concrete task instances with specific execution dates, mapped dependencies, and resource pools.
• Purpose: Enables efficient scheduling, parallelism, and state management during the actual run.