Skip to main content

AWS

When running pipelines on AWS, you might encounter the following issues.

Elastic Block Store (EBS)​

Volumes remain active after job completion​

On large AWS Batch clusters (hundreds of compute nodes or more), EC2 API rate limits can cause the automatic deletion of unattached EBS volumes to fail. Orphaned volumes that remain after jobs complete incur additional costs.

EBS autoscaling relies on an AWS-provided script on each container host that calls the EC2 API to delete each volume when its job finishes. When deletion fails, find orphaned volumes in the EC2 console or with a Lambda function and delete them manually. See Controlling your AWS costs by deleting unused Amazon EBS volumes.

Elastic Container Service (ECS)​

ECS agent Docker image pull frequency​

When Batch Forge creates an AWS Batch environment, it sets the ECS agent's ECS_IMAGE_PULL_BEHAVIOUR in the EC2 launch template:

  • Seqera Enterprise v22.01 or later: once
  • Seqera Enterprise v21.12 or earlier: default

See the AWS ECS documentation for the difference between these values.

note

This behavior can't be changed within Seqera Platform.

Container errors​

CannotPullContainerError … "Too Many Requests (HAP429)"​

CannotPullContainerError: Error response from daemon: error parsing HTTP 429 response body: invalid character 'T' looking for beginning of value: "Too Many Requests (HAP429)"

This error occurs when you exceed Docker Hub's rate limit of 100 anonymous pulls per 6 hours.

To resolve, add the following to your launch template:

echo ECS_IMAGE_PULL_BEHAVIOR=once >> /etc/ecs/ecs.config

CannotInspectContainerError​

Essential container in task exited - CannotInspectContainerError: Could not transition to inspecting; timed out after waiting 30s

To resolve:

  1. Upgrade your ECS agent to 1.54.1 or later. See Check for ECS Container Instance Agent Version to check your version.
  2. Provision more storage for your EC2 instance, preferably with EBS autoscaling for scalability.
  3. If the error includes command exit status: 123 and a permissions-denied error on a system command, make the ECS agent binary executable (chmod u+x).

Queues​

Distribute tasks across multiple AWS Batch queues​

You can identify only a single work queue when you define an AWS Batch compute environment, but you can distribute tasks across multiple queues in your pipeline configuration. Add a snippet like the following to your nextflow.config, or the Advanced options > Nextflow config file field of the launch form, to distribute processes across two queues by name:

# nextflow.config

process {
  withName: foo {
    queue: `TowerForge-1jJRSZmHyrrCvCVEOhmL3c-work`
  }
}

process {
  withName: bar {
    queue: `custom-second-queue`
  }
}

GPUs​

CUDA safe call. System has unsupported display driver / CUDA driver combination​

CUDA safe call. System has unsupported display driver / CUDA driver combination exiting

This error occurs when the container's CUDA runtime is newer than the NVIDIA driver on the compute environment's AMI.

To resolve, do one of the following:

  • Update the AMI to one with a newer NVIDIA driver. Use the latest AWS-recommended GPU-optimized ECS AMI (the default when Enable GPUs is set), or build a custom AMI with a driver version that meets the container's CUDA requirement. See the NVIDIA CUDA compatibility matrix for the minimum driver version.
  • Pin the container to a supported CUDA version. Use a container image built against a CUDA runtime the installed driver supports. NVIDIA Parabricks, for example, publishes image tags for each CUDA version. Select one that matches the AMI's driver.

To confirm the active driver on a failed task, see the Driver version field in GPU metrics.

Spot instances​

Tasks fail with exit code 143, or no exit code, and the log contains Host EC2 (instance i-xxxxxxxxx) terminated

AWS reclaimed the Spot instance running the task. See Manage AWS Spot interruptions for retry and fallback strategies.

Storage​

Write to S3 buckets that enforce AES256 server-side encryption​

note

Requires Seqera v21.10.4 and Nextflow 22.04.0 or later.

To save files to an S3 bucket with a policy that enforces AES256 server-side encryption, configure the nf-launcher script that invokes the Nextflow head job:

  1. Add the following to the Advanced options > Nextflow config file field of the Launch Pipeline screen:

    aws {
      client {
        storageEncryption = 'AES256'
      }
    }

  2. Add the following to the Advanced options > Pre-run script field:

    export TOWER_AWS_SSE=AES256

Tasks fail with S3 AccessDenied on Intelligent Compute​

A task fails at task runtime with AccessDenied on a bucket the pipeline could read or write before.

This issue occurs when the compute environment's Additional S3 buckets list does not cover every bucket the pipeline uses. Once the list is non-empty, Intelligent Compute scopes the IAM role it creates for the compute environment to the listed buckets plus the work directories, and denies everything else. A bucket encrypted with a customer-managed KMS key, or owned by another AWS account that has not granted access, fails the same way.

Resolution: add the failing bucket to Additional S3 buckets on the compute environment, or clear the list to restore account-wide S3 access for that compute environment. Either change applies to the next run — runs in progress keep the permissions they started with. For buckets in another AWS account, and for the encryption and access limitations, see S3 bucket access.

Runs or Studio sessions fail with S3 AccessDenied on a Seqera Compute additional bucket​

A pipeline run or a Studio session fails with AccessDenied on a bucket listed in the Additional S3 buckets of a Seqera Compute environment.

This issue occurs when the bucket policy in your AWS account does not grant access to the roles of this compute environment. Seqera Platform does not validate access to additional buckets when you create the compute environment or launch a run.

To resolve, check the following in your AWS account:

  • The bucket policy is applied to the right bucket, and its role ARNs match the Bucket policy section of this compute environment. The roles include the compute environment ID. A policy copied from another compute environment does not work.
  • If the entry has a path, the pipeline or data link only uses objects under that path.
  • For an SSE-KMS bucket, the key policy grants the roles kms:Decrypt and kms:GenerateDataKey*.
  • No Deny statement in the bucket policy matches the compute environment roles.

See Use S3 buckets in your own AWS account.

On this Page