Skip to main content

AWS Batch

The awsbatch executor runs pipeline tasks using AWS Batch, a managed compute service that runs containerized workloads in AWS. It submits each task as an AWS Batch job.

To enable this executor, set process.executor = 'awsbatch' in the nextflow.config file. You can launch the pipeline from a local machine or an EC2 instance. Every pipeline process must specify a container image using the container directive. The pipeline work directory must be in an S3 bucket.

Use the following process directives to control resource requests and other job characteristics:

The following hints are supported:

  • consumableResources: Specify AWS Batch consumable resources as a list of name-value pairs. For example:

    hints consumableResources: ['my-license-a': 1, 'my-license-b': 2]

See Amazon S3 for how to configure AWS credentials and S3 access. See the aws config scope for the full list of configuration options.

tip

For automated AWS infrastructure setup, consider using Seqera Platform, which can create AWS Batch compute environments for you.

Getting started​

  1. In the AWS Console, navigate to AWS Batch and create a compute environment (CE).

    1. If you use a custom AMI, specify the AMI ID in the CE configuration.
    2. Select an AMI (custom or existing) with Docker installed.
    3. Attach the AmazonS3FullAccess policy, which grants access to S3 buckets, to the instance role configured for the CE.
    4. If you plan to pull container images from Amazon Elastic Container Registry (ECR), also attach the AmazonEC2ContainerServiceforEC2Role policy to the instance role.

  2. In the AWS Console, create at least one job queue and bind it to the compute environment.

  3. In the AWS Console, create an S3 bucket for the work directory. You can also create separate buckets for input data and results, as needed.

  4. Make sure that every process in your pipeline specifies a container image with the container directive.

  5. Make sure that your container images are published in a container registry that AWS Batch can reach, such as Docker Hub, Quay, or Elastic Container Registry.

  6. Configure your AWS credentials. See Authentication for details.

  7. Add the following settings to your Nextflow configuration:

    • Set process.executor to awsbatch.
    • Set the AWS Batch job queue with the queue directive.
    • Set aws.region to the region of your AWS Batch environment.
    • Set workDir to an S3 path, for example, s3://<bucket>/work.

  8. Launch your pipeline:

    nextflow run <pipeline-name>

For example:

process {
    executor = 'awsbatch'
    queue = '<queue-name>'
    container = 'quay.io/biocontainers/salmon'
}

workDir = 's3://<bucket>/work'

aws.region = 'us-east-1'
warning

The work directory must contain at least one subdirectory. Do not use the bucket name alone, for example, s3://<bucket>.

tip

Nextflow automatically transfers any input data not stored in an S3 bucket to the pipeline work directory. Use this feature with caution to avoid unnecessary data transfers.

tip

To assign a different queue to each process, use the queue directive in the process definition or process selectors in your Nextflow configuration.

Required permissions​

AWS uses IAM policies to define permissions for IAM identities. To access AWS services, attach the required policies to the identity associated with your AWS credentials.

The following permissions are required to use AWS Batch, in addition to the S3 permissions described in Amazon S3:

  • To submit and manage Batch jobs:

    "batch:CancelJob"
    "batch:DescribeComputeEnvironments"
    "batch:DescribeJobDefinitions"
    "batch:DescribeJobQueues"
    "batch:DescribeJobs"
    "batch:ListJobs"
    "batch:RegisterJobDefinition"
    "batch:SubmitJob"
    "batch:TagResource"
    "batch:TerminateJob"

  • To view EC2 instances:

    "ec2:DescribeInstanceAttribute"
    "ec2:DescribeInstances"
    "ec2:DescribeInstanceStatus"
    "ec2:DescribeInstanceTypes"
    "ecs:DescribeContainerInstances"
    "ecs:DescribeTasks"

  • To pull container images from ECR repositories:

    "ecr:BatchCheckLayerAvailability"
    "ecr:BatchGetImage"
    "ecr:DescribeImages"
    "ecr:DescribeImageScanFindings"
    "ecr:DescribeRepositories"
    "ecr:GetAuthorizationToken"
    "ecr:GetDownloadUrlForLayer"
    "ecr:GetLifecyclePolicy"
    "ecr:GetLifecyclePolicyPreview"
    "ecr:GetRepositoryPolicy"
    "ecr:ListImages"
    "ecr:ListTagsForResource"

    Alternatively, use the AWS managed AmazonEC2ContainerRegistryReadOnly policy.

note

If you run Fargate or Fargate Spot, you might also need the following policies:

"ec2:DescribeSubnets"
"ecs:CreateCluster"
"ecs:DeleteCluster"
"ecs:DescribeClusters"
"ecs:ListClusters"

AWS CLI​

tip

The recommended approach for running Nextflow pipelines with AWS Batch is to use Wave and Fusion for S3 access instead of the AWS CLI. With this approach, you do not need to install and configure the AWS CLI in containers or custom AMIs.

Nextflow uses the AWS command line tool (aws) to stage input and output files between S3 and the task containers.

To make the aws command available, install it either in the container images or in a custom AMI that you use instead of the default AMI when you configure AWS Batch. See Custom AMIs for AWS Batch for how to create a custom AMI with the AWS CLI.

When the AWS CLI is installed in a custom AMI, set the aws.batch.cliPath option in your Nextflow configuration. For example:

aws.batch.cliPath = '/home/ec2-user/miniconda/bin/aws'

Replace the path with the location of the aws tool in your AMI.

warning

The grandparent directory of the aws tool is mounted into the container at the same path as on the host, for example, /home/ec2-user/miniconda. The mount shadows any existing files at that path in the container. Use a path that does not already exist in the container.

Container options​

Use the containerOptions directive to control the properties of the container execution associated with each Batch job.

The following container options are supported:

-e, --env string
    Set environment variables (format: <name> or <name>=<value>)
--init
    Run an init inside the container that forwards signals and reaps processes
--memory-swap int
    The total amount of swap memory (in MiB) the container can use: '-1' to enable unlimited swap
--memory-swappiness int
    Tune container memory swappiness (0 to 100) (default -1)
--privileged
    Give extended privileges to the container
--read-only
    Mount the container's root filesystem as read only
--shm-size int
    Size (in MiB) of /dev/shm
--tmpfs string
    Mount a tmpfs directory (format: <path>:<options>,size=<int>), size is in MiB
-u, --user string
    Username or UID (format: <name|uid>[:<group|gid>])
--ulimit string
    Ulimit options (format: <type>=<soft limit>[:<hard limit>])

You can pass container options in long form (for example, --option value) or, where available, short form (for example, -o value).

For example:

process.containerOptions = '--tmpfs /run:rw,noexec,nosuid,size=128 --tmpfs /app:ro,size=64'
process.containerOptions = '-e MYVAR1 --env MYVAR2=foo2 --env MYVAR3=foo3 --memory-swap 3240000 --memory-swappiness 20 --shm-size 16000000'
process.containerOptions = '--ulimit nofile=1280:2560 --ulimit nproc=16:32 --privileged'

See the AWS documentation for more details.

Custom job definition​

Nextflow automatically creates the Batch job definitions needed to execute tasks in your pipeline. You do not need to define them beforehand.

You might still need a custom job definition to fine-tune the configuration of a specific job, for example, to define custom mount paths.

To use a custom job definition, create it in the AWS Console (or by other means) and note its name. Then set the container directive to the job definition name prefixed by job-definition://, instead of a container image name:

process.container = 'job-definition://your-job-definition-name'

Volume mounts​

Use the aws.batch.volumes config option to mount host paths into task containers:

aws {
    region = 'eu-west-1'
    batch {
        volumes = '/tmp'
    }
}

To specify multiple volumes, use a list of paths or a comma-separated string. Use the standard Docker volume mount syntax for volumes where the container path differs from the host path or the volume is read-only. For example:

aws {
    region = 'eu-west-1'
    batch {
        volumes = ['/tmp', '/host/path:/mnt/path:ro']
    }
}

This example defines two volume mounts for the jobs in your pipeline. The first volume mounts the host path /tmp to the same path in the container, with the read-write access mode. The second volume mounts the host path /host/path to /mnt/path in the container, with the read-only access mode.

AWS Fargate​

Added in version 24.04

Nextflow provides experimental support for running AWS Batch jobs with Fargate resources.

You can use AWS Fargate with AWS Batch to run containers without managing servers or EC2 instances. With Fargate, you do not provision, configure, or scale clusters of virtual machines to run containers.

To use AWS Fargate in your pipeline, add the following settings to your nextflow.config file:

process.executor = 'awsbatch'
process.queue = '<aws-batch-queue>'
aws.region = '<aws-region>'
aws.batch.platformType = 'fargate'
aws.batch.jobRole = '<job-role-arn>'
aws.batch.executionRole = '<execution-role-arn>'
wave.enabled = true

See the AWS documentation for how to create the required AWS Batch queue for Fargate, the Batch job role, and the Batch execution role.

note

Nextflow uses s5cmd to download task inputs and upload task outputs. This capability requires Wave, which you enable in the Nextflow configuration as shown in the previous example. See Wave for more details.

Spot instances​

To use Spot Instances, create an AWS Batch compute environment with the SPOT provisioning model and bind it to the job queue used by your pipeline. No additional Nextflow configuration is required.

Spot Instances can be reclaimed by AWS before the job completes. Set aws.batch.maxSpotAttempts to have AWS Batch automatically retry a job when its instance is reclaimed:

aws.batch.maxSpotAttempts = 3

The default is 0 (no automatic retry), or 5 when Fusion snapshots are enabled. AWS Batch does not retry jobs that fail for other reasons. The Nextflow errorStrategy directive handles those failures.

Fusion file system​

The AWS Batch executor supports Fusion. With Fusion, tasks use Amazon S3 as a virtual distributed file system. Fusion optimizes data transfer and speeds up most task I/O operations. With Fusion, the container images and the AMI do not need the AWS CLI.

See AWS Batch for more information about configuring Fusion for AWS Batch.

Troubleshooting​

Problem: The pipeline run terminates with an AWS error message similar to the following:

JobQueue <queue> not found

Make sure you define an AWS region in the Nextflow configuration file and that it matches the region where you created your Batch environment.

Problem: A task fails with the following error message:

Process <task> terminated for an unknown reason -- Likely it has been terminated by the external system

This error can occur when Batch cannot execute the process script. A common cause is a Docker container image with a non-standard entrypoint that prevents execution of the Bash launcher script Nextflow uses to run the job.

It can also occur if the AWS CLI does not run correctly.

Other places to check for error information:

  • The .nextflow.log file.
  • The job execution log in the AWS Batch dashboard.
  • The CloudWatch logs found in the /aws/batch/job log group.

Problem: A task is stuck in the RUNNABLE status and the pipeline output is similar to the following:

executor >  awsbatch (1)
process > <process> (1) [  0%] 0 of ....

The pipeline can hang indefinitely when a job is held in the queue and never runs. In the AWS Console, the queue reports the job as RUNNABLE, but the job never leaves that status.

This issue has several possible causes. Most relate to the compute environment workload or configuration, the Docker service or container configuration, or network status.

See this AWS page for ways to investigate and work around the issue.

On this Page