Nextflow DSL2

DSL2 replaces monolithic scripts with reusable, namespaced modules that encapsulate individual processes. Each module can declare its own process, workflow, and executor directives.

• Modules enable code reuse across pipelines and teams
• Each module can specify its own container image via the container directive
• Modules are imported using include { PROCESS } from './module.nf' syntax
• Supports versioned module repositories for pipeline reproducibility

DSL2 formalizes asynchronous, reactive data streams through typed channels that connect processes without explicit file path management.

• Queue channels: FIFO queues for standard process-to-process communication
• Value channels: Singleton values broadcast to all downstream consumers
• Channels support operators like .map(), .filter(), .mix(), and .collect()
• Implicit parallelization occurs when multiple items flow through a channel simultaneously

Every process in DSL2 can declare its own Docker or Singularity container, ensuring identical runtime environments across local workstations, HPC clusters, and cloud VMs.

• Use container 'biocontainers/samtools:1.15' to pin exact tool versions
• Singularity/Apptainer support enables HPC deployments without Docker daemon privileges
• Conda environments also supported via conda directive for non-containerized execution
• Eliminates the 'works on my machine' reproducibility crisis in bioinformatics

DSL2 allows nested workflow definitions that can be composed into higher-order pipelines, treating entire sub-workflows as reusable components.

• Define sub-workflows with workflow ALIGN { ... } syntax
• Compose them: workflow { ALIGN(); CALL_VARIANTS() }
• Sub-workflows accept and emit channels, enabling clean interface contracts
• Enables domain-specific pipeline libraries shared across research groups

DSL2 separates pipeline logic from execution infrastructure through a pluggable executor model that supports AWS Batch, Google Cloud Life Sciences, Kubernetes, SLURM, and more.

• Switch from local to cloud execution by changing a single executor directive
• Autoscaling support for cloud executors dynamically provisions compute resources
• Native integration with AWS S3 and Google Cloud Storage for data staging
• Fusion file system enables direct cloud object store access without local downloads

Nextflow's continuation semantics track every task's completion state, allowing failed pipelines to resume from the point of failure without recomputing successful steps.

• -resume flag reuses cached results from previous runs
• Task hashing detects changes in code, inputs, or containers to invalidate cache
• Work directory stores intermediate files for debugging failed tasks
• Publish directory directive copies final outputs to a clean, organized location

The fundamental unit of code organization in DSL2. A module is a self-contained script that defines a specific process, allowing it to be versioned, tested, and reused independently.

• process definition: Contains the input, output, and script directives for a single tool.
• Implicit workflow: Each module file can contain a single workflow definition that is executed by default.
• Portability: Modules can be imported from local paths, remote Git repositories, or community hubs like nf-core.

A composition layer that orchestrates the invocation of modules and sub-workflows. DSL2 separates the definition of a task from the logic that connects them.

• workflow keyword: Defines a named, callable block that takes data channels as input and emits processed channels.
• Explicit data flow: Channels are passed between processes and sub-workflows using function-like syntax, making dependencies visually clear.
• Nesting: Workflows can be nested within other workflows, enabling the construction of complex, hierarchical pipeline topologies.

Asynchronous, first-in-first-out queues that connect processes and carry data through the pipeline. They are the backbone of Nextflow's reactive dataflow programming model.

• value channels: Bind a single value, which can be read multiple times without being consumed.
• queue channels: A non-blocking channel that allows processes to consume items as they become available.
• Operators: A rich library of operators (map, filter, join, collect) transforms channel contents without blocking execution.

The mechanism that guarantees computational reproducibility. DSL2 pipelines define the exact software environment for each process using containers, eliminating dependency conflicts.

• Docker: The most common container runtime, specified with the container directive in a process.
• Singularity/Apptainer: The standard for HPC clusters, automatically converted from Docker images by Nextflow.
• Conda: A non-container alternative for managing binary dependencies, specified with the conda directive.

The abstraction layer that allows a single DSL2 pipeline to run on diverse compute platforms without modification. The executor handles job submission and monitoring.

• Local: Executes jobs on the submitting machine, ideal for development and testing.
• SLURM / Grid Engine: Submits jobs to HPC workload managers, the backbone of institutional bioinformatics.
• AWS Batch / Google Cloud: Provisions cloud compute instances on-demand, enabling infinite horizontal scaling.

A community-driven initiative that provides a curated collection of gold-standard, peer-reviewed DSL2 pipelines. It is the definitive source for production-ready bioinformatics workflows.

• Standardized structure: All pipelines follow a strict template, ensuring consistency and reducing onboarding time.
• Parameter schema: A JSON-based schema validates user inputs, preventing runtime errors from misconfigured parameters.
• Tower integration: nf-core pipelines are designed to be launched and monitored seamlessly via Nextflow Tower.