Intelligent Compute
Seqera Intelligent Compute is in preview. Seqera must enable it on a per-workspace basis before you can use it. Contact your account manager to request access for one or more workspaces.
By design, Intelligent Compute overrides the CPU and memory your pipeline's process directives request. It sizes each task from that task's history in previous runs, and picks the most cost-effective instance type for the predicted resources. A task can therefore run with less, or more, CPU and memory than its directives specify.
Intelligent Compute is a scheduling service that runs Nextflow pipelines on Seqera-managed virtual machines (VMs) in your own cloud account. It allocates compute resources based on what each task needs rather than what the pipeline requests, packs tasks onto a cluster of VMs that scales beyond a single instance, and uses Spot capacity where it can. This reduces cost and improves utilization across a run.
The standard cloud compute environments (AWS Cloud, Azure Cloud, and Google Cloud) run each pipeline on a single VM with a local executor. When you select Seqera Intelligent Compute as the Compute mode of one of these compute environments, Nextflow submits each task to the Intelligent Compute scheduler instead. The scheduler selects instance types, launches and terminates VMs, places tasks on them, and retries tasks that are interrupted.
Get started
Select your cloud provider for setup steps, required permissions, and the resources Seqera creates in your account:
| Cloud provider | Compute environment type | Guide |
|---|---|---|
| AWS | AWS Cloud | Intelligent Compute on AWS |
| Azure | Azure Cloud | Intelligent Compute on Azure |
| Google Cloud | Google Cloud | Intelligent Compute on Google Cloud |
All clouds share the following requirements:
- Seqera has enabled Intelligent Compute for your workspace.
- Nextflow 26.04 or later. Nextflow 26.10 or later is recommended, and Seqera pre-fills a 26.10 release in the launch form when the pipeline has no saved version. API and CLI launches that don't set a version use Nextflow 26.04. See Nextflow version.
- Wave and Fusion. Seqera enables both automatically.
The compute environment's head-job fields, such as Instance type and Boot disk size, apply only to the VM that runs the Nextflow head job. To restrict the instance types for tasks, use Compute instance types.
Feature availability by cloud
| Feature | AWS | Azure | Google Cloud |
|---|---|---|---|
| Spot, Spot first, and On-demand provisioning | Yes | Yes | Yes |
| Fusion snapshots | Yes | No | Yes |
| Warm pool | Yes | No | No |
| Cost reporting | AWS Cost Explorer | Azure Cost Management | BigQuery billing export |
| Default On-Demand fallback cost limit | 3.25 times the Spot price | No limit | 2.48 times the Spot price |
| Spot interruption notice | 2 minutes | About 30 seconds | Up to 150 seconds |
Configuration options
When you select Seqera Intelligent Compute as the Compute mode, an Intelligent Compute options section appears on the compute environment form. The defaults in the following table are the values the form pre-fills.
| Option | Values | Default | Clouds | What you can do with it |
|---|---|---|---|---|
| Provisioning model | Spot first, Spot, On-demand | Spot first | All | Choose how VMs are procured. Spot first uses Spot and falls back to On-Demand, subject to the cost limit. Spot never falls back: the task fails instead. On-demand uses On-Demand only. See Spot and On-Demand capacity. |
| Max spot attempts | 1–10 | 5 | All | Set how many times a task can run on Spot before it moves to On-Demand (Spot first) or fails (Spot). Each Spot interruption uses one attempt. Hidden for On-demand. |
| Prediction model | qr/v3 | qr/v3 | All | The model that predicts each task's CPU and memory from its previous runs. Use qr/v3. |
| Fusion snapshots | Enabled / Disabled | AWS: Enabled. Google Cloud: Disabled | AWS, Google Cloud | Resume an interrupted Spot task from a checkpoint instead of restarting it. Not supported on Azure, and not available with On-demand. See Retries and Fusion snapshots. |
| Fast instance storage | Enabled / Disabled | Disabled | All | Restrict selection to instance types with local NVMe or SSD storage, for I/O-heavy pipelines. Fewer instance types can raise cost or reduce Spot availability. |
| Compute instance types | List of instance types | Empty | All | Restrict the instance types the scheduler can select. When empty, the scheduler picks the most cost-effective type for each task. |
| Max CPUs per user | Integer | Empty (no limit) | All | Cap the vCPUs one user's runs can use at the same time on this compute environment. Tasks over the cap aren't rejected: Nextflow submits them, and they stay in Submitted until the user's running tasks free enough vCPUs. |
| BigQuery billing export table | project.dataset.table | Empty | Google Cloud | Report run cost. See Cost reporting. |
| Enable warm pool | Enabled / Disabled | Disabled | AWS | Keep idle VMs running to reduce task start latency. AWS bills idle VMs to your account. Requires Compute instance types. |
| Desired VMs | Integer | 1 | AWS | Set how many idle VMs the warm pool keeps. |
| Scale to zero | Seconds | 0 (never) | AWS | Release the warm pool's idle VMs after this many seconds without activity. |
Retries and Fusion snapshots
A task can be retried at two independent layers:
- The scheduler retries an interrupted task inside the same Nextflow task attempt. Neither Nextflow nor Seqera Platform sees these retries. Each Spot interruption uses one of the task's Max spot attempts.
- Nextflow retries a task only after the scheduler reports it as failed, consuming one
process.maxRetriesattempt.
Without Fusion snapshots, an interrupted task restarts from the beginning on its new VM. With snapshots, Fusion checkpoints the task when the interruption notice arrives, and the task resumes from that checkpoint.
The following diagram shows one task with Fusion snapshots enabled. Without snapshots, including on Azure, which doesn't support them, each run starts the task from the beginning:
- A snapshot doesn't use a Spot attempt. The interruption already used one. If a snapshot is missing or incomplete, the task restarts from the beginning instead.
- Snapshots restore onto compatible instance types only: the same family, or one with a compatible, newer CPU architecture. This also applies to the On-Demand fallback.
- Snapshots must finish within the interruption notice: 2 minutes on AWS, and up to 150 seconds on Google Cloud. Tasks with a large memory footprint may not checkpoint in time. See Fusion snapshots.
Nextflow retries
When the scheduler runs out of options, it reports the task to Nextflow as failed, with a message such as Task preempted by cloud provider (spot instance reclaimed). With errorStrategy = 'retry', Nextflow resubmits the task. The resubmitted task is new to the scheduler: its Spot attempts start again, and under Spot first it starts on Spot again.
Max spot attempts is separate from the aws.batch.maxSpotAttempts Nextflow configuration property, which applies to AWS Batch only.
Spot and On-Demand capacity
Spot VMs cost much less than On-Demand VMs, but the cloud provider can reclaim them at short notice, and Spot capacity is sometimes unavailable. The provisioning model decides what the scheduler does:
| Situation | Spot first | Spot | On-demand |
|---|---|---|---|
| Spot VM reclaimed, Spot attempts left | Retry on a new Spot VM | Retry on a new Spot VM | — |
| Spot VM reclaimed, no Spot attempts left | Move to On-Demand, if the cost limit allows | Fail the task | — |
| No Spot capacity at launch | Launch On-Demand instead, if the cost limit allows | Try other instance types, then fail | — |
| VM lost for another reason | Retry on On-Demand | Retry on On-Demand | Retry on On-Demand |
The following swimlane shows how a task moves between Spot and On-Demand under Spot first. A task that completes at any step ends there. Each new VM can be a different instance type from the one before. When the cost limit refuses On-Demand, the scheduler first retries on Spot with newly selected instance types, and the task fails only if Spot capacity is still unavailable:
- A task never moves from On-Demand back to Spot. It stays on On-Demand for the rest of that Nextflow attempt. Other tasks in the run keep starting on Spot.
- The On-Demand VM can be a different instance type. The scheduler launches the cheapest available On-Demand VM from the instance types it selected for the task, which can be a different type or family from the Spot VM the task last ran on.
On-Demand fallback cost limit
With Spot first, the scheduler falls back to On-Demand only while On-Demand doesn't exceed the Spot price by more than a set limit. It compares the highest On-Demand hourly price with the highest Spot hourly price across the instance types it selected for the task. The defaults are 3.25 times the Spot price on AWS, 2.48 times on Google Cloud, and no limit on Azure. Seqera can change these defaults as prices change.
The cost limit can prevent a Spot first task from ever moving to On-Demand. When On-Demand exceeds the limit, the scheduler selects new instance types and retries on Spot. If Spot capacity is still unavailable, the task fails, even though On-Demand capacity was available. The error message starts with on-demand fallback refused and states the ratio of the On-Demand price to the Spot price.
To change the limit for all tasks in a run, set seqera.executor.machineRequirement.onDemandCostRatio in the pipeline's Nextflow configuration (Nextflow 26.10 or later). Set it to 0 to remove the limit, so tasks always fall back to On-Demand whatever the price:
seqera.executor.machineRequirement.onDemandCostRatio = 2.5
To override the limit for a single process, use the seqera/machineRequirement.onDemandCostRatio process hint. See On-demand fallback cost limit in the Nextflow documentation.
To never fall back to On-Demand, use the Spot provisioning model. If a pipeline must complete even when Spot capacity is scarce, use On-demand, or set the cost limit to 0.
Per-user CPU cap
Max CPUs per user caps the vCPUs a single user can hold at once on a shared compute environment. Every user gets the same budget. One user's runs cannot take over the whole compute environment.
The scheduler applies the cap when it submits each task:
- Usage is the total vCPUs of the user's submitted and running tasks across their runs on the compute environment.
- A task that would exceed the cap waits until enough of the user's tasks complete.
- A task that alone requests more than the cap fails with
Task requires <n> vCPUs but the per-user maximum for this compute environment is <max> vCPUs. Lower itscpusdirective or raise the cap.
The cap must be a positive integer. Seqera rejects 0 or lower with Max CPUs per user must be a positive integer.
When set, Seqera adds seqera.executor.schedulingRequirement.maxCpusPerUser = <n> to each run's generated Nextflow configuration. With Nextflow 26.08.0-edge or later, a pipeline can set the same option to lower its own cap. The scheduler clamps a higher value to the compute environment's cap.
Runs take the cap in effect when they start. Changing it applies to new runs only.
Resource metrics
The Metrics tab for a run on Intelligent Compute shows three resource values for CPU and memory:
| Metric | Source | What it represents |
|---|---|---|
| Requested | Pipeline process directives | The CPU and memory your pipeline asked for. |
| Allocated | Scheduler decision | The CPU and memory the scheduler assigned to the task container. Differs from Requested when the scheduler picks a more cost-effective shape that still meets the task's needs. |
| Used | Nextflow trace data | The CPU and memory the task consumed (pcpu × realtime for CPU, peakRss for memory). Absent for tasks that didn't produce trace data. |