Local execution
With Fusion, you can run Nextflow pipelines using the local executor and a cloud storage bucket as the pipeline scratch directory. This is useful to scale your pipeline execution vertically with a large compute instance, without the need to allocate a large storage volume for temporary pipeline data.
Nextflow CLI
This configuration requires Docker or a similar container engine to run pipeline tasks.
- AWS S3
- Azure Blob Storage
- Google Cloud Storage
- MinIO
- Oracle Object Storage
-
Set
AWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
environment variables to grant Nextflow and Fusion access to your storage credentials. See Credentials for more information. -
Add the following to your
nextflow.config
file:wave.enabled = true
docker.enabled = true
fusion.enabled = true
fusion.exportStorageCredentials = true -
Run the pipeline with the Nextflow run command:
nextflow run <PIPELINE_SCRIPT> -w s3://<S3_BUCKET>/work
Replace the following:
PIPELINE_SCRIPT
: your pipeline Git repository URIS3_BUCKET
: your S3 bucket name
To achieve optimal performance, set up an SSD volume as the temporary directory.
The option fusion.exportStorageCredentials
leaks credentials on the task launcher script created by Nextflow.
This option should only be used for testing and development purposes.
-
Set
AZURE_STORAGE_ACCOUNT_NAME
andAZURE_STORAGE_ACCOUNT_KEY
orAZURE_STORAGE_SAS_TOKEN
environment variables to grant Nextflow and Fusion access to your storage credentials. See Credentials for more information. -
Add the following to your
nextflow.config
file:wave.enabled = true
docker.enabled = true
fusion.enabled = true
fusion.exportStorageCredentials = true -
Run the pipeline with the Nextflow run command:
nextflow run <PIPELINE_SCRIPT> -w az://<BLOB_STORAGE>/scratch
Replace the following:
<PIPELINE_SCRIPT>
: your pipeline Git repository URI<BLOB_STORAGE>
: your Azure Blob Storage
To achieve optimal performance, set up an SSD volume as the temporary directory.
The option fusion.exportStorageCredentials
leaks credentials on the task launcher script created by Nextflow.
This option should only be used for testing and development purposes.
-
Set the
GOOGLE_APPLICATION_CREDENTIALS
environment variable with your service account JSON key to grant Nextflow and Fusion access to your storage credentials. See Credentials for more information. -
Add the following to your
nextflow.config
file:wave.enabled = true
docker.enabled = true
fusion.enabled = true
fusion.exportStorageCredentials = true -
Run the pipeline with the Nextflow run command:
nextflow run <PIPELINE_SCRIPT> -w gs://<GCS_BUCKET>/work
Replace the following:
<PIPELINE_SCRIPT>
: your pipeline Git repository URI<GCS_BUCKET>
: your Google Cloud Storage bucket to which you have read-write access
To achieve optimal performance, set up an SSD volume as the temporary directory.
The option fusion.exportStorageCredentials
leaks credentials on the task launcher script created by Nextflow.
This option should only be used for testing and development purposes.
-
Run a local instance of MinIO using Docker:
docker run -p 9000:9000 \
--rm -d -p 9001:9001 \
-e "MINIO_ROOT_USER=admin" \
-e "MINIO_ROOT_PASSWORD=secret" \
quay.io/minio/minio server /data --console-address ":9001" -
Open the MinIO console
http://localhost:9001
-
Create storage credentials and a bucket.
-
Set
AWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
environment variables to grant Nextflow and Fusion access to your storage credentials. See Credentials for more information. -
Add the following to your
nextflow.config
file:aws.client.endpoint = 'http://localhost:9000'
aws.client.s3PathStyleAccess = true
wave.enabled = true
docker.enabled = true
fusion.enabled = true
fusion.exportStorageCredentials = true -
Run the pipeline with the Nextflow run command:
nextflow run <PIPELINE_SCRIPT> -w s3://<S3_BUCKET>/work
Replace the following:
PIPELINE_SCRIPT
: your pipeline Git repository URIS3_BUCKET
: your S3 bucket name
To achieve optimal performance, set up an SSD volume as the temporary directory.
The option fusion.exportStorageCredentials
leaks credentials on the task launcher script created by Nextflow.
This option should only be used for testing and development purposes.
Oracle Object Storage relies on the S3-like API compatibility provided by Oracle storage and not by native Nextflow and Fusion support. It may not support all Nextflow and Fusion features.
-
Set
AWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
environment variables to grant Nextflow and Fusion access to your storage credentials. -
Add the following to your
nextflow.config
file:aws.region = '<AWS_REGION>'
aws.client.endpoint = 'https://<BUCKET_NAMESPACE>.compat.objectstorage.<AWS_REGION>.oraclecloud.com'
aws.client.protocol = 'https'
aws.client.signerOverride = 'AWSS3V4SignerType'
aws.client.s3PathStyleAccess = true
wave.enabled = true
docker.enabled = true
docker.containerOptions = '-e FUSION_AWS_REGION=<AWS_REGION>'
fusion.enabled = true
fusion.exportStorageCredentials = trueReplace the following:
<AWS_REGION>
: your AWS region<BUCKET_NAMESPACE>
: your bucket name
-
Run the pipeline with the Nextflow run command:
nextflow run <PIPELINE_SCRIPT> -w s3://<BUCKET>/work
Replace the following:
PIPELINE_SCRIPT
: your pipeline Git repository URIBUCKET
: your bucket
To achieve optimal performance, set up an SSD volume as the temporary directory.
The option fusion.exportStorageCredentials
leaks credentials on the task launcher script created by Nextflow.
This option should only be used for testing and development purposes.