AWS Batch

The awsbatch executor runs pipeline tasks using AWS Batch, a managed compute service that runs containerized workloads in AWS. It submits each task as an AWS Batch job.

To enable this executor, set process.executor = 'awsbatch' in the nextflow.config file. You can launch the pipeline from a local machine or an EC2 instance. Every pipeline process must specify a Docker image using the container directive. The pipeline work directory must be located in an S3 bucket.

Use the following process directives to control resource requests and other job characteristics:

• accelerator
• arch (only when using Fargate platform type for AWS Batch)
• container
• containerOptions
• cpus
• disk (only when using Fargate platform type for AWS Batch)
• memory
• queue
• resourcelabels
• time

The following hints are supported:

• consumableResources: Specify AWS Batch consumable resources as a list of name-value pairs. For example:

  hints consumableResources: ['my-license-a': 1, 'my-license-b': 2]

See AWS Batch for more information.