Workflow Definition Language (WDL)

A WDL uses a declarative syntax, meaning developers specify what the workflow should accomplish—its tasks, data dependencies, and desired outputs—rather than how to execute each step procedurally. The workflow engine interprets this declaration to create an optimal execution plan. This abstraction separates business logic from orchestration mechanics, enhancing portability and readability.

• Example: Defining a task that calls a machine learning model without writing the code to manage its execution environment or retry logic.
• Contrast: Imperative code (e.g., in Python) explicitly codes the sequence of operations and error handling.

A core characteristic is the clear separation between tasks (discrete units of work) and the workflow that composes them. A task defines a single operation, such as running a script or calling an API, including its runtime requirements (container image, CPU, memory). The workflow defines the graph of these tasks, their data flow, and control logic. This modularity allows for task reuse across different workflows and independent testing.

• Task Properties: Command to run, input/output declarations, resource requirements, and meta sections.
• Workflow Properties: Calls to tasks, conditional logic (if statements), and scatter operations for parallel processing.

WDLs enforce explicit data flow through strongly typed inputs and outputs for every task and workflow. Data dependencies between tasks are defined by connecting one task's output to another's input, creating a Directed Acyclic Graph (DAG). The engine uses this graph to schedule tasks only when their inputs are ready. Strong typing (e.g., File, String, Int, Array[Float]) prevents runtime errors and ensures data integrity.

• Data Plumbing: Output of Task A (output { File results_csv }) is wired directly to the input of Task B (input { File data_file }).
• Type Safety: The engine validates all data assignments before execution, catching mismatches early.

A well-designed WDL is portable and runtime-agnostic. The workflow definition does not hardcode details of the execution environment (e.g., cloud provider, cluster scheduler). Instead, it declares abstract resource needs. A runtime configuration file, separate from the WDL, maps these abstractions to concrete backend systems (e.g., Google Cloud Life Sciences, AWS Batch, Kubernetes). This allows the same WDL to run identically across different infrastructures, a key requirement for reproducible computational pipelines.

• Separation of Concerns: The what (WDL) is separate from the where and with-what (runtime config).
• Example: A task requesting cpu: 2 can be satisfied by a Docker container on a local machine or a virtual machine in the cloud.

Reliable WDLs provide native syntax for fault tolerance patterns, reducing boilerplate code. Key constructs include:

• Retry Policies: Define automatic retries for a failed task with configurable delay (e.g., exponential backoff) and exit code handling.
• Runtime Attributes: Specify disk space, memory, and boot disk size to prevent infrastructure-related failures.
• Conditional Execution: Use if and scatter blocks to create dynamic, data-dependent execution paths. These features move error handling from the task implementation into the orchestration layer, making workflows more robust and declarative.

A mature WDL has a standardized structure (typically JSON or a human-readable derivative like WDL 1.0/2.0) that enables a rich ecosystem of developer tooling. This includes:

• Linters and Validators: Static analysis tools to check syntax and type correctness before runtime.
• Documentation Generators: Auto-generate documentation from code comments within the WDL.
• Graph Visualization: Tools to render the workflow DAG from the WDL source.
• Language Server Protocols: IDE support for autocomplete, syntax highlighting, and error checking. This ecosystem reduces developer friction and enforces best practices across teams.

A workflow engine is the core runtime software that interprets a Workflow Definition Language (WDL) and manages the execution of workflows. It is responsible for:

• State Management: Tracking the progress of each process instance.
• Task Invocation: Calling the code or service for each defined activity.
• Control Flow: Enforcing the sequence, conditional branching, and parallel execution as specified in the WDL.
• Reliability: Implementing retry logic, checkpointing, and failure handling. Popular examples include Apache Airflow, Temporal, and AWS Step Functions.

A Directed Acyclic Graph (DAG) is the most common computational model underlying a WDL. It represents a workflow where:

• Nodes are individual tasks or activities.
• Directed Edges represent dependencies and data flow between tasks.
• Acyclic means there are no circular dependencies, ensuring tasks can be executed in a finite, logical order. This structure allows the workflow engine to calculate an execution plan, identify tasks that can run in parallel, and detect invalid cycles during compilation. An Airflow DAG is a canonical example.

A state machine is a model for defining workflows where execution is represented as a series of states and the transitions between them. Key characteristics include:

• Finite States: The workflow can only be in one of a predefined set of states (e.g., RUNNING, WAITING, SUCCEEDED).
• Event-Driven Transitions: Moves between states are triggered by events or the completion of actions.
• Deterministic Logic: The next state is determined solely by the current state and the incoming event. This model is foundational to declarative orchestration tools like AWS Step Functions, where a workflow is defined as a Step Functions state machine using JSON.

The Saga pattern is a critical design pattern for managing long-running, distributed transactions within a workflow, especially when a WDL coordinates microservices. It addresses the lack of distributed transactions by:

• Sequencing Local Transactions: Breaking the overall business process into a series of local, committed transactions.
• Compensating Transactions: Providing a corresponding rollback operation (a compensating transaction) for each committed step.
• Orchestration or Choreography: Can be managed centrally by the workflow engine (orchestration) or via events between services (choreography). This pattern is essential for building reliable, multi-step processes like order fulfillment that must maintain data consistency across services.

Event-driven orchestration is a paradigm where workflow execution is initiated and progressed by events rather than a rigid, pre-defined script. In this model:

• Triggers: Workflows or tasks start in response to events like a file landing in cloud storage, an API call, or a message on a queue.
• Loose Coupling: The workflow engine reacts to the event stream, enabling dynamic, reactive systems.
• Decoupled Tasks: Tasks often communicate asynchronously via a task queue or event bus. This approach contrasts with purely time-scheduled workflows (using a cron trigger) and is key for building responsive, scalable integrations and data pipelines.

Workflow-as-Code is the practice of defining, versioning, and testing workflows using standard software development tools and lifecycle practices. Instead of configuring workflows in a GUI, they are authored in code (e.g., Python, YAML, Java). Benefits include:

• Version Control: Workflow definitions are stored in Git, enabling collaboration, code review, and rollback.
• CI/CD Integration: Workflows can be tested and deployed as part of an automated pipeline.
• Reusability & Modularity: Code abstractions (functions, classes, modules) can be used to create reusable workflow components.
• Declarative vs. Imperative: Can support both declarative orchestration (specifying what) and imperative logic (specifying how). Tools like Apache Airflow and Temporal exemplify this paradigm.