AWS Batch
Fusion Snapshots enable checkpoint/restore functionality for Nextflow processes running on AWS Batch Spot instances. When a Spot instance interruption occurs, AWS provides a guaranteed 120-second warning window to checkpoint and save the task state before the instance terminates.
For retry strategies and where Seqera Platform shows evidence of an interruption, see AWS Spot interruption management.
Seqera Platform compute environment requirements
Fusion Snapshots require the following Seqera Platform compute environment configuration:
- Provider: AWS Batch
- Work directory: S3 bucket in the same region as compute resources
- Fusion snapshots: Enabled
- Config mode: Batch Forge
- Provisioning model: Spot. Do not enable Fusion Snapshots on an On-Demand compute environment.
- AMI: See Selecting an AMI for details
- Instance types: Leave empty to use the recommended sizes, or set your own under Advanced options. See Selecting an EC2 instance.
Fusion Snapshots work with sensible defaults (e.g., 5 automatic retry attempts). For configuration options, see Advanced configuration.
Selecting an AMI
Fusion Snapshots require instances running Amazon Linux 2023 (which ships with Linux Kernel 6.1) and an ECS container-optimized AMI for optimal performance.
Seqera Cloud
Seqera Cloud AWS Batch compute environments use an ECS container-optimized AMI by default. No additional AMI configuration is required.
Seqera Enterprise
Specify an Amazon Linux 2023 ECS-optimized AMI for your region when creating your compute environment.
To find the recommended AMI:
-
Retrieve the application configuration:
export REGION=<AWS_REGION>
aws ssm get-parameter --name "/aws/service/ecs/optimized-ami/amazon-linux-2023/recommended" --region $REGIONReplace
<AWS_REGION>with your AWS region (for example,eu-central-1).The output for the
eu-central-1region is similar to the following:{
"Parameter": {
"Name": "/aws/service/ecs/optimized-ami/amazon-linux-2023/recommended",
"Type": "String",
"Value": "{\"ecs_agent_version\":\"1.88.0\",\"ecs_runtime_version\":\"Docker version 25.0.6\",\"image_id\":\"ami-0281c9a5cd9de63bd\",\"image_name\":\"al2023-ami-ecs-hvm-2023.0.20241115-kernel-6.1-x86_64\",\"image_version\":\"2023.0.20241115\",\"os\":\"Amazon Linux 2023\",\"schema_version\":1,\"source_image_name\":\"al2023-ami-minimal-2023.6.20241111.0-kernel-6.1-x86_64\"}",
"Version": 61,
"LastModifiedDate": "2024-11-18T17:08:46.926000+01:00",
"ARN": "arn:aws:ssm:eu-central-1::parameter/aws/service/ecs/optimized-ami/amazon-linux-2023/recommended",
"DataType": "text"
} -
Identify the
image_idin your output (e.g,ami-0281c9a5cd9de63bdin the above example) and set in the Advanced options > AMI ID field when you create your Seqera compute environment.
Selecting an EC2 instance
AWS provides a guaranteed 120-second reclamation window. Checkpoint time is primarily determined by memory usage. Other factors like the number of open file descriptors also affect performance.
When you create a Batch Forge compute environment with Fusion snapshots enabled and leave Instance types empty, Seqera Platform restricts the compute queue to the recommended sizes that your AWS region offers:
c6id.4xlarge,c6id.8xlarge,c6id.12xlarge,m6id.4xlarge,m6id.8xlarge,r6id.2xlarge
Platform sets these sizes on the AWS Batch compute environment it creates, so Instance types stays empty in the Seqera compute environment. If you set Instance types yourself, Platform uses your list instead. If your region offers none of these sizes, for example London (eu-west-2), Platform falls back to its standard instance type selection. In that case, set Instance types yourself.
Platform applies the recommended sizes when it creates the compute environment. Compute environments created before this behavior was available, or with Fusion Snapshots enabled after creation (on the compute environment, a pipeline, or a run), keep their instance types. Seqera Enterprise 26.2 and earlier do not set these sizes. In those cases, create a new compute environment and set the instance types yourself:
- Copy the comma-separated list above.
- In Advanced options, paste the list into Instance types. See AWS Batch (Cloud) or AWS Batch (Enterprise).
- Enable Fusion snapshots and set Provisioning model to Spot.
The Instance types field accepts families or specific sizes. Enter specific sizes, not family names such as c6id, m6id, or r6id. A family permits every size in that family, including sizes that miss the 120-second window. Do not use default_x86_64.
If the compute environment is not restricted to suitable instance types, AWS Batch can schedule tasks on instance types with burst-only ("up to") network bandwidth, no NVMe storage (no d suffix), ARM64 architecture, or a memory:bandwidth ratio worse than 5:1. These types cannot transfer checkpoint data within the 120-second window.
Fusion takes snapshots only on Spot instances, even when the option is enabled. Do not enable Fusion Snapshots on an On-Demand compute environment.
When you select instance types:
- Select instances with guaranteed network bandwidth, not "up to" values.
- Maintain a 5:1 or better ratio between memory (GiB) and network bandwidth (Gbps). Instances with lower ratios complete snapshots faster.
- Prefer NVMe storage instances (those with a
dsuffix:c6id,r6id,m6id). - Use
x86_64instances for incremental snapshots. Do not enable Use Graviton CPU architecture.
Do not add FPGA instance types (f1, f2) to Instance types. Fusion Snapshots cannot restore a task on a reclaimed FPGA instance. Run FPGA workloads such as Illumina DRAGEN in an On-Demand compute environment. See Hardware compatibility.
For example, a c6id.8xlarge instance provides 64 GiB memory and 12.5 Gbps guaranteed network bandwidth. This configuration can transfer the entire memory contents to S3 in approximately 70 seconds. Instances with memory:bandwidth ratios over 5:1 may not complete a full-memory transfer before termination.
| Instance type | Cores | Memory (GiB) | Network bandwidth (Gbps) | Memory:bandwidth ratio | Estimated snapshot time |
|---|---|---|---|---|---|
c6id.4xlarge | 16 | 32 | 12.5 | 2.56:1 | ~45 seconds |
c6id.8xlarge | 32 | 64 | 12.5 | 5.12:1 | ~70 seconds |
r6id.2xlarge | 8 | 64 | 12.5 | 5.12:1 | ~70 seconds |
m6id.4xlarge | 16 | 64 | 12.5 | 5.12:1 | ~70 seconds |
c6id.12xlarge | 48 | 96 | 18.75 | 5.12:1 | ~70 seconds |
m6id.8xlarge | 32 | 128 | 12.5 | 10.24:1 | ~105 seconds |
At 10.24:1, m6id.8xlarge can miss the 120-second window when a task uses most of its 128 GiB. Prefer a type at or below 5:1, or set process.resourceLimits to cap task memory so snapshots complete within the window.
Incremental snapshots are enabled by default on x86_64 instances.
Resource limits
A single job can request more resources than are available on a single instance. To prevent this, set resource limits using the process.resourceLimits directive in your Nextflow configuration. See Resource limits for more information.
Storage and cleanup
Fusion writes snapshot data to a .fusion/dump/ directory inside each task work directory in your S3 work bucket. AWS bills it as standard S3 storage. Fusion Snapshots do not use S3 object versioning.
When a task process exits, Fusion removes the memory image files from that task's checkpoints, including checkpoints written by earlier attempts of the same task. Small metadata and log files remain. Snapshot data has no expiry. Anything left behind by a run that was killed before its tasks could exit stays in the bucket until you delete it.
Check the work directories of runs that ended abnormally for leftover .fusion/dump/ directories, and remove them if you no longer need the diagnostic data. See Snapshot storage and lifecycle for details.