Intelligent Compute on AWS

Preview

Seqera Intelligent Compute is in preview. Seqera must enable it on a per-workspace basis before you can use it. Contact your account manager to request access for one or more workspaces.

On AWS, Intelligent Compute runs on an AWS Cloud compute environment. The scheduler launches your pipeline tasks on EC2 instances in your AWS account, as Spot or On-Demand instances according to the provisioning model. For how scheduling, retries, and the Spot to On-Demand fallback work, see Intelligent Compute.

Created resources

Seqera creates and manages the following resources in your AWS account:

• EC2 instances that run your tasks, with EBS scratch volumes on instance types that have no NVMe storage. Terminated when the cluster scales down.
• IAM role and instance profile (seqera-sched-vm-instance-*, seqera-sched-vm-profile-*), one per compute environment and Additional S3 buckets list. Seqera doesn't currently remove unused ones, so delete them yourself if you need to.
• Security group seqera-sched-vm, egress only, one per VPC, unless the compute environment specifies its own security groups.
• S3 gateway VPC endpoint, when the compute environment specifies no subnets and the VPC has none.
• CloudWatch log groups /seqera/scheduler/vm/<cluster-id> for tasks and /seqera/platform for the head job. Seqera doesn't remove these, so set a retention policy if you need one.

IAM permissions

Attach two IAM policies to the IAM user or role that Seqera uses to access your AWS account:

1. AWS Cloud policy: required for all AWS Cloud compute environments. If you already use an AWS Cloud compute environment, it's already in place.
2. Intelligent Compute policy: the additional permissions the scheduler needs.

AWS Cloud policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AwsCloudCreate",
      "Effect": "Allow",
      "Action": [
        "iam:CreateRole",
        "iam:AddRoleToInstanceProfile",
        "iam:CreateInstanceProfile",
        "iam:AttachRolePolicy",
        "iam:PutRolePolicy",
        "iam:TagRole",
        "iam:TagInstanceProfile"
      ],
      "Resource": [
        "arn:aws:iam::*:role/TowerForge-*",
        "arn:aws:iam::*:instance-profile/TowerForge-*"
      ]
    },
    {
      "Sid": "AwsCloudCreatePassRole",
      "Effect": "Allow",
      "Action": [
        "iam:PassRole"
      ],
      "Resource": "arn:aws:iam::*:role/TowerForge-*"
    },
    {
      "Sid": "AwsCloudLaunchEC2",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateTags",
        "ec2:DeleteTags",
        "ec2:DescribeInstances",
        "ec2:RunInstances",
        "ec2:TerminateInstances"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AwsCloudLaunchLogs",
      "Effect": "Allow",
      "Action": [
        "logs:GetLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:log-group:*:log-stream:*"
    },
    {
      "Sid": "AwsCloudLaunchS3",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AwsCloudDelete",
      "Effect": "Allow",
      "Action": [
        "iam:GetRole",
        "iam:ListAttachedRolePolicies",
        "iam:ListRolePolicies",
        "iam:DeleteRole",
        "iam:DeleteInstanceProfile",
        "iam:RemoveRoleFromInstanceProfile",
        "iam:DetachRolePolicy",
        "iam:DeleteRolePolicy"
      ],
      "Resource": [
        "arn:aws:iam::*:role/TowerForge-*",
        "arn:aws:iam::*:instance-profile/TowerForge-*"
      ]
    },
    {
      "Sid": "AwsCloudRead",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstanceTypes",
        "ec2:DescribeKeyPairs",
        "ec2:DescribeVpcs",
        "ec2:DescribeImages",
        "ec2:DescribeSubnets",
        "ec2:DescribeSecurityGroups",
        "s3:ListAllMyBuckets"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AwsCloudUserdataCheck",
      "Effect": "Allow",
      "Action": [
        "ec2:GetConsoleOutput"
      ],
      "Resource": "*"
    },
    {
      "Sid": "OptionalLineageIntegrationSNSAndS3",
      "Effect": "Allow",
      "Action": [
        "sns:CreateTopic",
        "sns:SetTopicAttributes",
        "sns:Subscribe",
        "sns:ConfirmSubscription",
        "sns:Unsubscribe",
        "sns:DeleteTopic",
        "s3:CreateBucket",
        "s3:GetBucketNotification",
        "s3:PutBucketNotification",
        "s3:GetBucketLocation",
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:sns:*:*:seqera-lineage-*",
        "arn:aws:s3:::seqera-lineage-*",
        "arn:aws:s3:::seqera-lineage-*/*"
      ]
    }
  ]
}

Download aws-cloud-full-policy.json

Intelligent Compute policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "IAMRoleManagement",
      "Effect": "Allow",
      "Action": [
        "iam:CreateRole",
        "iam:GetRole",
        "iam:DeleteRole",
        "iam:PutRolePolicy",
        "iam:DeleteRolePolicy",
        "iam:ListRolePolicies",
        "iam:AttachRolePolicy",
        "iam:DetachRolePolicy",
        "iam:ListAttachedRolePolicies",
        "iam:CreateInstanceProfile",
        "iam:GetInstanceProfile",
        "iam:DeleteInstanceProfile",
        "iam:AddRoleToInstanceProfile",
        "iam:RemoveRoleFromInstanceProfile"
      ],
      "Resource": [
        "arn:aws:iam::*:role/seqera-sched-*",
        "arn:aws:iam::*:instance-profile/seqera-sched-*"
      ]
    },
    {
      "Sid": "PassRoleToEC2",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "arn:aws:iam::*:role/seqera-sched-*",
      "Condition": {
        "StringEquals": {
          "iam:PassedToService": "ec2.amazonaws.com"
        }
      }
    },
    {
      "Sid": "ServiceLinkedRoles",
      "Effect": "Allow",
      "Action": "iam:CreateServiceLinkedRole",
      "Resource": "arn:aws:iam::*:role/aws-service-role/*",
      "Condition": {
        "StringEquals": {
          "iam:AWSServiceName": "spot.amazonaws.com"
        }
      }
    },
    {
      "Sid": "CloudWatchLogs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:DeleteLogGroup",
        "logs:DescribeLogGroups",
        "logs:TagResource",
        "logs:DescribeLogStreams",
        "logs:GetLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:log-group:/seqera/*"
    },
    {
      "Sid": "EC2NetworkDiscovery",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeVpcs",
        "ec2:DescribeSubnets",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeRouteTables",
        "ec2:DescribeVpcEndpoints",
        "ec2:DescribeInstances",
        "ec2:CreateSecurityGroup",
        "ec2:CreateVpcEndpoint",
        "ec2:AuthorizeSecurityGroupEgress",
        "ec2:CreateTags"
      ],
      "Resource": "*"
    },
    {
      "Sid": "VMEC2Operations",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstanceTypes",
        "ec2:RunInstances",
        "ec2:TerminateInstances",
        "ec2:RevokeSecurityGroupEgress",
        "ec2:DeleteSecurityGroup"
      ],
      "Resource": "*"
    },
    {
      "Sid": "S3Access",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": "*"
    },
    {
      "Sid": "CostExplorer",
      "Effect": "Allow",
      "Action": "ce:GetCostAndUsage",
      "Resource": "*"
    }
  ]
}

Download aws-cloud-intelligent-compute-policy.json

You can omit the following statements of the Intelligent Compute policy:

• ServiceLinkedRoles: if the AWSServiceRoleForEC2Spot role already exists in your account, or you don't use Spot.
• CostExplorer: if you don't need run cost. See Cost reporting.

If your account encrypts EBS volumes by default with a customer-managed KMS key, also grant kms:CreateGrant, kms:GenerateDataKeyWithoutPlaintext, kms:Decrypt, and kms:DescribeKey on that key.

To create the policies and the IAM user or role, see Create the IAM policy and AWS credential options.

note

If you set up Intelligent Compute before it moved to EC2 instances, your existing policy may include ECS, ECR, Auto Scaling, and SSM permissions, and ec2:AuthorizeSecurityGroupIngress. Intelligent Compute no longer uses them, and you can remove them. Compute environments that ran on Amazon ECS may also have left seqera-sched-* ECS clusters, Auto Scaling groups, and launch templates, and a seqera-sched-ecs security group in the default VPC, in your account. Seqera no longer deletes these resources, so remove them manually.

Create the compute environment

Prerequisites

You need the following:

• Seqera has enabled Intelligent Compute for your workspace.
• AWS credentials in Seqera with the AWS Cloud and Intelligent Compute policies attached.
• An S3 bucket for the work directory, in the region where you run pipelines.

1. In your Seqera workspace, select Compute Environments, then select Add compute environment.
2. Enter a name and select AWS Cloud as the platform.
3. Select your AWS credentials.
4. Select the Region where Seqera launches compute resources.
5. Enter a Work directory (for example, s3://my-bucket/work).
6. Under Compute mode, select Seqera Intelligent Compute.
7. Under Intelligent Compute options, review the defaults. See Configuration options.
8. Optional: Enter the Additional S3 buckets your pipelines read and write. See S3 bucket access.
9. Optional: Under Advanced options, configure networking.
10. Select Add.

Seqera creates the IAM role, instance profile, and EC2 instances when you launch the first pipeline.

Use Graviton CPU architecture under Advanced options applies only to the instance that runs the Nextflow head job, not to the instances that run your tasks.

Networking

Task instances have no inbound rules. The scheduler agent on each instance connects out to Seqera over HTTPS, so the subnets need outbound access to Seqera, Amazon S3, Wave, and your container registries.

VPC ID	Subnets	Result
Not set	Not set	The region's default VPC. Seqera creates an S3 gateway endpoint if the VPC has none.
Set	Not set	All subnets of the VPC. Seqera creates an S3 gateway endpoint if the VPC has none.
Set	Set	Only the listed subnets. A private subnet must already reach S3 through a gateway endpoint or a NAT gateway.

Subnets in several Availability Zones give the scheduler more places to find Spot capacity. The instance that runs the Nextflow head job tries the listed subnets in order, moving to the next one when a subnet or its Availability Zone has no capacity; see Multiple subnets. If you set Security groups, they replace seqera-sched-vm. They must be in the same VPC as the subnets and allow outbound HTTPS and DNS.

S3 bucket access

The IAM role that Intelligent Compute attaches to task instances gets its S3 permissions from the compute environment's Additional S3 buckets list:

• Empty list (default): S3 access account-wide.
• Populated list: access to the listed buckets only, plus the compute environment and run work directories, which Seqera adds automatically.

On each bucket, the role can read, write, delete, and tag objects, and list the bucket.

caution

A populated Additional S3 buckets list is the complete allow-list. A pipeline that touches a bucket missing from the list fails with AccessDenied at task runtime, not at validation or launch. List every bucket your pipelines use, including reference data (for example, s3://ngi-igenomes) and the outdir.

Entries can be s3:// URIs or bare bucket names. Seqera ignores key prefixes: s3://shared-bucket/team-a grants access to all of shared-bucket. Changes apply to the next run.

To use a bucket owned by another AWS account, add it to Additional S3 buckets, and ask the bucket owner to grant your account access in the bucket policy. To name the Intelligent Compute role, match it with a pattern, so the policy keeps working if the role is recreated with a new internal ID:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "SeqeraIntelligentCompute",
      "Effect": "Allow",
      "Principal": "*",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:GetObjectTagging", "s3:PutObjectTagging", "s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": ["arn:aws:s3:::reference-data-shared", "arn:aws:s3:::reference-data-shared/*"],
      "Condition": {
        "ArnLike": { "aws:PrincipalArn": "arn:aws:iam::111122223333:role/seqera-sched-vm-instance-*" }
      }
    }
  ]
}

Replace reference-data-shared with the bucket name and 111122223333 with your compute account ID. For read-only data, drop the Put and Delete actions.

Limitations

• Buckets encrypted with a customer-managed KMS key (SSE-KMS) aren't supported. SSE-S3 works.
• Seqera grants access per bucket, not per prefix.
• The role can't abort multipart uploads. Add an S3 lifecycle rule that expires incomplete multipart uploads.

For AccessDenied errors, see Tasks fail with S3 AccessDenied on Intelligent Compute.

Cost reporting

Seqera shows each run's cost on the run details page, from AWS Cost Explorer, once AWS publishes the billing data, usually 24–48 hours after the run. Seqera finds a run's cost by the seqera.io:platform:workflowId tag, which Intelligent Compute applies to every EC2 instance and EBS volume it launches for the run.

Tags on task instances

Intelligent Compute tags each task instance and its EBS volumes with the following keys. AWS tag keys can't contain /, so Seqera writes each / as :.

Tag key	Value
seqera.io:platform:workflowId	Seqera run ID
seqera.io:platform:workspaceId	Seqera workspace ID. Nextflow 26.10 or later.
seqera.io:platform:computeEnvId	Seqera compute environment ID. Nextflow 26.10 or later.
nextflow.io:projectName	Pipeline project name, for example nf-core/rnaseq
nextflow.io:runName	Nextflow run name
nextflow.io:sessionId	Nextflow session ID
nextflow.io:userName	Seqera user who launched the run. Before Nextflow 26.10, the operating system user that runs Nextflow.
nextflow.io:revision	Pipeline revision
nextflow.io:commitId	Pipeline commit ID
nextflow.io:repository	Pipeline repository URL
nextflow.io:manifestName	Pipeline manifest name
nextflow.io:runtimeVersion	Nextflow version
nextflow.io:resume	true if the run was resumed, otherwise false

Task instances also carry:

• The run's resource labels, including dynamic resource labels such as platformRun=${workflowId}.
• Labels set with a fixed value at the top level of your Nextflow configuration, for example process.resourceLabels = [team: 'genomics'].

One instance runs tasks from many processes, so it can't carry a label whose value differs between tasks. Labels set with a closure, such as process.resourceLabels = { [step: task.process] }, apply to tasks only, not to instances.

Set up cost reporting

1. Make sure the IAM policy attached to your credentials includes the CostExplorer statement (ce:GetCostAndUsage). The Intelligent Compute policy includes it. Without it, Seqera can't read run cost.

2. Launch a run, so that the seqera.io:platform:workflowId tag exists on at least one resource. The key can take up to 24 hours to appear in the AWS Billing and Cost Management console. This first run shows no cost in Seqera, because you activate the tag after it ran.

3. In the management (payer) account, activate seqera.io:platform:workflowId as a user-defined cost allocation tag: in the AWS Billing and Cost Management console, select Cost allocation tags, search for the key, select it, and select Activate. To activate it with the AWS CLI from the management account:

   aws ce update-cost-allocation-tags-status \
     --cost-allocation-tags-status '[{"TagKey":"seqera.io:platform:workflowId","Status":"Active"}]' \
     --region us-east-1

4. Optional: activate the other tag keys and resource label keys you want to group by in Cost Explorer.

Cost Explorer ignores tags that aren't activated, and attributes cost to a tag only from the time you activate it. Runs before activation show no cost in Seqera.

What run cost includes

Run cost includes the task instances and their EBS volumes. It doesn't include:

• The head job instance, which runs Nextflow. Seqera doesn't currently capture the head job's cost for Intelligent Compute runs.
• The warm pool: with Enable warm pool, tasks run on warm-pool instances, which don't carry the run's tags, so neither run cost nor reports filtered by your resource labels include their cost.
• Other services, such as S3 storage and requests, and CloudWatch Logs.