Nextflow
"Dataflow variables are spectacularly expressive in concurrent programming"
Henri E. Bal , Jennifer G. Steiner , Andrew S. Tanenbaum
Many scientific and data-intensive analyses are a chain of command-line tools that pass data to each other as files, rather than through databases or APIs. These pipelines run over thousands of inputs, each step can take hours, and each tool needs specific software versions. Running that reliably means moving files to where the work runs, splitting the work into parallel jobs, submitting them to a cluster or cloud, retrying failures, and keeping track of what has already finished. Nextflow handles this for you. You write each step in any language, and Nextflow stages each task's input files into its own working directory, whether they are stored locally or in cloud object storage. It runs every file through the pipeline in parallel, can give each task its own container, and runs the same pipeline on a laptop, an HPC cluster, or the cloud with only a configuration change. There is no server to deploy, and if a run fails, -resume picks up where it stopped.
Nextflow is a workflow system for creating scalable, portable, and reproducible workflows. Its dataflow programming model lets you describe how data moves between steps, and Nextflow works out what can run in parallel. As well as containers, it supports package managers such as Conda and Spack for managing software.
Get started
To get started with Nextflow:
- See the Nextflow overview to learn key concepts
- Download and install Nextflow
- Set up an environment with the Nextflow VS Code extension
- Run your first script
To continue learning about Nextflow, visit the Nextflow community training portal and find a training course that is right for you. Seqera, the company that develops Nextflow, also runs a variety of training events. See Seqera Events for more information.
Community
You can post questions in the Nextflow community forum or the Nextflow Slack. Bugs and feature requests should be reported as GitHub issues.
The Nextflow community is highly active with regular community meetings, events, a podcast, and more. You can view this material on the Nextflow YouTube channel.
The nf-core project is a community effort aggregating high-quality Nextflow workflows that can be used by everyone.
Contributing
Contributions to Nextflow are welcome. See Contributing for details.
License
Nextflow is released under the Apache 2.0 license. Nextflow is a registered trademark.
Citations
If you use Nextflow in your work, please cite:
- P. Di Tommaso, et al. Nextflow enables reproducible computational workflows. Nature Biotechnology 35, 316–319 (2017) doi:10.1038/nbt.3820