Azure Batch

This guide assumes you already have an Azure account with a valid Azure Subscription. For details, visit Azure Free Account. Ensure you have sufficient permissions to create resource groups, an Azure Storage account, and an Azure Batch account.

Azure concepts

Regions

Azure regions are specific geographic locations around the world where Microsoft has established data centers to host its cloud services. Each Azure region is a collection of data centers that provide users with high availability, fault tolerance, and low latency for cloud services. Each region offers a wide range of Azure services that can be chosen to optimize performance, ensure data residency compliance, and meet regulatory requirements. Azure regions also enable redundancy and disaster recovery options by allowing resources to be replicated across different regions, enhancing the resilience of applications and data.

Resource groups

An Azure resource group is a logical container that holds related Azure resources such as virtual machines, storage accounts, databases, and more. A resource group serves as a management boundary to organize, deploy, monitor, and manage the resources within it as a single entity. Resources in a resource group share the same lifecycle, meaning they can be deployed, updated, and deleted together. This also enables easier access control, monitoring, and cost management, making resource groups a foundational element in organizing and managing cloud infrastructure in Azure.

Accounts

Azure uses accounts for each service. For example, an Azure Storage account will house a collection of blob containers, file shares, queues, and tables. An Azure subscription can have multiple Azure Storage and Azure Batch accounts - however, a Platform compute environment can only use one of each. Multiple Platform compute environments can be created to use separate credentials, Azure Storage accounts, and Azure Batch accounts.

Service principals

An Azure service principal is an identity created specifically for applications, hosted services, or automated tools to access Azure resources. It acts like a user identity with a defined set of permissions, enabling resources authenticated through the service principal to perform actions within the Azure account. The platform can utilize an Azure service principal to authenticate and access Azure Batch for job execution and Azure Storage for data management.

Create Azure resources

Resource group

Create a resource group to link your Azure Batch and Azure Storage account:

A resource group can be created while creating an Azure Storage account or Azure Batch account.

1. Log in to your Azure account, go to the Create Resource group page, and select Create new resource group.
2. Enter a name for the resource group, such as seqeracompute.
3. Choose the preferred region.
4. Select Review and Create to proceed.
5. Select Create.

Storage account

After creating a resource group, set up an Azure Storage account:

1. Log in to your Azure account, go to the Create storage account page, and select Create a storage account.

   If you haven't created a resource group, you can do so now.

2. Enter a name for the storage account, such as seqeracomputestorage.
3. Choose the preferred region. This must be the same region as the Batch account.
4. Platform supports all performance or redundancy settings — select the most appropriate settings for your use case.
5. Select Next: Advanced.
6. Enable storage account key access.
7. Select Next: Networking.
   • Enable public access from all networks. You can enable public access from selected virtual networks and IP addresses, but you will be unable to use Forge to create compute resources. Disabling public access is not supported.
8. Select Data protection.
   • Configure appropriate settings. All settings are supported by the platform.
9. Select Encryption.
   • Only Microsoft-managed keys (MMK) are supported.
10. In tags, add any required tags for the storage account.
11. Select Review and Create.
12. Select Create to create the Azure Storage account.
    • You will need at least one Blob Storage container to act as a working directory for Nextflow.
13. Go to your new storage account and select + Container to create a new Blob Storage container. A new container dialogue will open. Enter a suitable name, such as seqeracomputestorage-container.
14. Go to the Access Keys section of your new storage account (seqeracomputestorage in this example).
15. Store the access keys for your Azure Storage account, to be used when you create a Seqera compute environment.

Blob container storage credentials are associated with the Batch pool configuration. Avoid changing these credentials in your Seqera instance after you have created the compute environment.

Batch account

After you have created a resource group and Storage account, create a Batch account:

1. Log in to your Azure account and select Create a batch account on this page.
2. Select the existing resource group or create a new one.
3. Enter a name for the Batch account, such as seqeracomputebatch.
4. Choose the preferred region. This must be the same region as the Storage account.
5. Select Advanced.
6. For Pool allocation mode, select Batch service.
7. For Authentication mode, select Shared Key.
8. Select Networking. Ensure networking access is sufficient for Platform and any additional required resources.
9. Add any Tags to the Batch account, if needed.
10. Select Review and Create.
11. Select Create.
12. Go to your new Batch account, then select Access Keys.
13. Store the access keys for your Azure Batch account, to be used when you create a Seqera compute environment.

    A newly-created Azure Batch account may not be entitled to create virtual machines without making a service request to Azure. See Azure Batch service quotas and limits for more information.

14. Select the + Quotas tab of the Azure Batch account to check and increase existing quotas if necessary.
15. Select + Request quota increase and add the quantity of resources you require. Here is a brief guideline:
    • Active jobs and schedules: Each Nextflow process will require an active Azure Batch job per pipeline while running, so increase this number to a high level. See here to learn more about jobs in Azure Batch.
    • Pools: Each platform compute environment requires one Azure Batch pool. Each pool is composed of multiple machines of one virtual machine size.

    To use separate pools for head and compute nodes, see this FAQ entry.

    • Batch accounts per region per subscription: Set this to the number of Azure Batch accounts per region per subscription. Only one is required.
    • Spot/low-priority vCPUs: Platform does not support spot or low-priority machines when using Forge, so when using Forge this number can be zero. When manually setting up a pool, select an appropriate number of concurrent vCPUs here.
    • Total Dedicated vCPUs per VM series: See the Azure documentation for virtual machine sizes to help determine the machine size you need. We recommend the latest version of the ED series available in your region as a cost-effective and appropriately-sized machine for running Nextflow. However, you will need to select alternative machine series that have additional requirements, such as those with additional GPUs or faster storage. Increase the quota by the number of required concurrent CPUs. In Azure, machines are charged per cpu minute so there is no additional cost for a higher number.

Credentials

There are two types of Azure credentials available: access keys and Entra service principals.

Access keys are simple to use but have several limitations:

• Access keys are long-lived.
• Access keys provide full access to the Azure Storage and Azure Batch accounts.
• Azure allows only two access keys per account, making them a single point of failure.

Entra service principals are accounts which can be granted access to Azure Batch and Azure Storage resources:

• Service principals enable role-based access control with more precise permissions.
• Service principals map to a many-to-many relationship with Azure Batch and Azure Storage accounts.
• Some Azure Batch features are only available when using a service principal.

The two Azure credential types use different authentication methods. You can add more than one credential to a workspace, but Platform compute environments use only one credential at any given time. While separate credentials can be used by separate compute environments concurrently, they are not cross-compatible — access granted by one credential will not be shared with the other.

Access keys

Batch Forge compute environments must use access keys for authentication. Service principals are only supported in manual compute environments.

To create an access key:

1. Navigate to the Azure Portal and sign in.
2. Locate the Azure Batch account and select Keys under Account management. The Primary and Secondary keys are listed here. Copy one of the keys and save it in a secure location for later use.
3. Locate the Azure Storage account and, under the Security and Networking section, select Access keys. Key1 and Key2 options are listed here. Copy one of them and save it in a secure location for later use.
4. In your Platform workspace Credentials tab, select the Add credentials button and complete the following fields:
   • Enter a Name for the credentials
   • Provider: Azure
   • Select the Shared key tab
   • Add the Batch account and Blob Storage account names and access keys to the relevant fields.
5. Delete the copied keys from their temporary location after they have been added to a credential in Platform.

Entra service principal

Batch Forge compute environments must use access keys for authentication. Service principals are only supported in manual compute environments.

The use of Entra service principals in manual compute environments requires the use of a managed identity.

See Create a service principal for more details.

To create an Entra service principal:

1. In the Azure Portal, navigate to Microsoft Entra ID. Under App registrations, select New registration.
2. Provide a name for the application. The application will automatically have a service principal associated with it.
3. Assign roles to the service principal:
   1. Go to the Azure Storage account. Under Access Control (IAM), select Add role assignment.
   2. Select the Storage Blob Data Reader and Storage Blob Data Contributor roles.
   3. Select Members, then Select Members. Search for your newly created service principal and assign the role.
   4. Repeat the same process for the Azure Batch account, using the Azure Batch Contributor role.
4. Platform will need credentials to authenticate as the service principal:
   1. Navigate back to the app registration. On the Overview page, save the Application (client) ID value for use in Platform.
   2. Select Certificates & secrets, then New client secret. A new secret is created containing a value and secret ID. Save both values securely for use in Platform.
5. In your Platform workspace Credentials tab, select the Add credentials button and complete the following fields:
   • Enter a Name for the credentials
   • Provider: Azure
   • Select the Entra tab
   • Complete the remaining fields: Batch account name, Blob Storage account name, Tenant ID (Application (client) ID in Azure), Client ID (Client secret ID in Azure), Client secret (Client secret value in Azure).
6. Delete the ID and secret values from their temporary location after they have been added to a credential in Platform.

Platform compute environment

There are two ways to create an Azure Batch compute environment in Seqera Platform:

• Batch Forge: Automatically creates Azure Batch resources.
• Manual: For using existing Azure Batch resources.

Batch Forge

Batch Forge automatically creates resources that you may be charged for in your Azure account. See Cloud costs for guidelines to manage cloud resources effectively and prevent unexpected costs.

Create a Batch Forge Azure Batch compute environment:

1. In a workspace, select Compute Environments > New Environment.

2. Enter a descriptive name, such as Azure Batch (east-us).

3. Select Azure Batch as the target platform.

4. Choose existing Azure credentials or add a new credential.

   Batch Forge compute environments must use access keys for authentication. Entra service principals are only supported in manual compute environments.

5. Add the Batch account and Blob Storage account names and access keys.

6. Select a Region, such as eastus.

7. In the Pipeline work directory field, enter the Azure blob container created previously. For example, az://seqeracomputestorage-container/work.

   When you specify a Blob Storage bucket as your work directory, this bucket is used for the Nextflow cloud cache by default. You can specify an alternative cache location with the Nextflow config file field on the pipeline launch form.

8. Select Enable Wave containers to facilitate access to private container repositories and provision containers in your pipelines using the Wave containers service. See Wave containers for more information.

9. Select Enable Fusion v2 to allow access to your Azure Blob Storage data via the Fusion v2 virtual distributed file system. This speeds up most data operations. The Fusion v2 file system requires Wave containers to be enabled. See Fusion file system for configuration details.

   Use Fusion v2

   The compute recommendations below are based on internal benchmarking performed by Seqera. Benchmark runs of nf-core/rnaseq used profile test_full, consisting of an input dataset with 16 FASTQ files and a total size of approximately 123.5 GB.

   Azure virtual machines include fast SSDs and require no additional storage configuration for Fusion. For optimal performance, use VMs with sufficient local storage to support Fusion's streaming data throughput.

   1. Use Seqera Platform version 23.1 or later.
   2. Use an Azure Blob storage container as the pipeline work directory.
   3. Enable Wave containers and Fusion v2.
   4. Select the Batch Forge config mode.
   5. Specify suitable VM sizes under VMs type. A Standard_E16d_v5 VM or larger is recommended for production use.

   We recommend selecting machine types with a local temp storage disk of at least 200 GB and a random read speed of 1000 MBps or more for large and long-lived production pipelines. To work with files larger than 100 GB, increase temp storage accordingly (400 GB or more).

   The suffix d after the core number (e.g., Standard_E16*d*_v5) denotes a VM with a local temp disk. Select instances with Standard SSDs — Fusion does not support Azure network-attached storage (Premium SSDv2, Ultra Disk, etc.). Larger local storage increases Fusion's throughput and reduces the chance of overloading the machine. See Sizes for virtual machines in Azure for more information.

10. Set the Config mode to Batch Forge.

11. Enter the default VMs type, depending on your quota limits set previously. The default is Standard_D4_v3.

12. Enter the VMs count. If autoscaling is enabled (default), this is the maximum number of VMs you wish the pool to scale up to. If autoscaling is disabled, this is the fixed number of virtual machines in the pool.

13. Enable Autoscale to scale up and down automatically, based on the number of pipeline tasks. The number of VMs will vary from 0 to VMs count.

14. Enable Dispose resources for Seqera to automatically delete the Batch pool if the compute environment is deleted on the platform.

15. Select or create Container registry credentials to authenticate a registry (used by the Wave containers service). It is recommended to use an Azure Container registry within the same region for maximum performance.

16. Apply Resource labels. This will populate the Metadata fields of the Azure Batch pool.

17. Expand Staging options to include:

    • Optional pre- or post-run Bash scripts that execute before or after the Nextflow pipeline execution in your environment.
    • Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the Nextflow config file field in the pipeline launch form. These values can be overridden during pipeline launch.

    Configuration settings in this field override the same values in the pipeline repository nextflow.config file. See Nextflow config file for more information on configuration priority.

18. Specify custom Environment variables for the Head job and/or Compute jobs.

19. Configure any advanced options you need:

    • Use Jobs cleanup policy to control how Nextflow process jobs are deleted on completion. Active jobs consume the quota of the Azure Batch account. By default, jobs are terminated by Nextflow and removed from the quota when all tasks succesfully complete. If set to Always, all jobs are deleted by Nextflow after pipeline completion. If set to Never, jobs are never deleted. If set to On success, successful tasks are removed but failed tasks will be left for debugging purposes.
    • Use Token duration to control the duration of the SAS token generated by Nextflow. This must be as long as the longest period of time the pipeline will run.

20. Select Add to finalize the compute environment setup. It will take a few seconds for all the resources to be created before the compute environment is ready to launch pipelines.

See Launch pipelines to start executing workflows in your Azure Batch compute environment.

Manual

This section is for users with a pre-configured Azure Batch pool. This requires an existing Azure Batch account with an existing pool.

Your Seqera compute environment uses resources that you may be charged for in your Azure account. See Cloud costs for guidelines to manage cloud resources effectively and prevent unexpected costs.

Create a manual Seqera Azure Batch compute environment:

1. In a workspace, select Compute Environments > New Environment.

2. Enter a descriptive name for this environment, such as Azure Batch (east-us).

3. For Provider, select Azure Batch.

4. Select your existing Azure credentials (access keys or Entra service principal) or select + to add new credentials.

   To authenticate using an Entra service principal, you must include a user-assigned managed identity. See Managed identity below.

5. Select a Region, such as eastus (East US).

6. In the Pipeline work directory field, add the Azure blob container created previously. For example, az://seqeracomputestorage-container/work.

   When you specify a Blob Storage bucket as your work directory, this bucket is used for the Nextflow cloud cache by default. You can specify an alternative cache location with the Nextflow config file field on the pipeline launch form.

7. Select Enable Wave containers to facilitate access to private container repositories and provision containers in your pipelines using the Wave containers service. See Wave containers for more information.

8. Select Enable Fusion v2 to allow access to your Azure Blob Storage data via the Fusion v2 virtual distributed file system. This speeds up most data operations. The Fusion v2 file system requires Wave containers to be enabled. See Fusion file system for configuration details.

   Use Fusion v2

   The compute recommendations below are based on internal benchmarking performed by Seqera. Benchmark runs of nf-core/rnaseq used profile test_full, consisting of an input dataset with 16 FASTQ files and a total size of approximately 123.5 GB.

   Azure virtual machines include fast SSDs and require no additional storage configuration for Fusion. For optimal performance, use VMs with sufficient local storage to support Fusion's streaming data throughput.

   1. Use Seqera Platform version 23.1 or later.
   2. Use an Azure Blob storage container as the pipeline work directory.
   3. Enable Wave containers and Fusion v2.
   4. Specify suitable VM sizes under VMs type. A Standard_E16d_v5 VM or larger is recommended for production use.

   We recommend selecting machine types with a local temp storage disk of at least 200 GB and a random read speed of 1000 MBps or more for large and long-lived production pipelines. To work with files larger than 100 GB, increase temp storage accordingly (400 GB or more).

   The suffix d after the core number (e.g., Standard_E16*d*_v5) denotes a VM with a local temp disk. Select instances with Standard SSDs — Fusion does not support Azure network-attached storage (Premium SSDv2, Ultra Disk, etc.). Larger local storage increases Fusion's throughput and reduces the chance of overloading the machine. See Sizes for virtual machines in Azure for more information.

9. Set the Config mode to Manual.

10. Enter the Compute Pool name. This is the name of the Azure Batch pool you created previously in the Azure Batch account.

    The default Azure Batch implementation uses a single pool for head and compute nodes. To use separate pools for head and compute nodes (for example, to use low-priority VMs for compute jobs), see this FAQ entry.

11. Enter a user-assigned Managed identity client ID, if one is attached to your Azure Batch pool. See Managed Identity below.

12. Apply Resource labels. This will populate the Metadata fields of the Azure Batch pool.

13. Expand Staging options to include:

    • Optional pre- or post-run Bash scripts that execute before or after the Nextflow pipeline execution in your environment.
    • Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the Nextflow config file field in the pipeline launch form. These values can be overridden during pipeline launch.

    Configuration settings in this field override the same values in the pipeline repository nextflow.config file. See Nextflow config file for more information on configuration priority.

    To use managed identities, Platform requires Nextflow version 24.06.0-edge or later. Add export NXF_VER=24.06.0-edge to the Global Nextflow config field for your compute environment to use this Nextflow version by default.

14. Define custom Environment Variables for the Head Job and/or Compute Jobs.

15. Configure any necessary advanced options:

    • Use Jobs cleanup policy to control how Nextflow process jobs are deleted on completion. Active jobs consume the quota of the Azure Batch account. By default, jobs are terminated by Nextflow and removed from the quota when all tasks succesfully complete. If set to Always, all jobs are deleted by Nextflow after pipeline completion. If set to Never, jobs are never deleted. If set to On success, successful tasks are removed but failed tasks will be left for debugging purposes.
    • Use Token duration to control the duration of the SAS token generated by Nextflow. This must be as long as the longest period of time the pipeline will run.

16. Select Add to complete the compute environment setup. The creation of resources will take a few seconds, after which you can launch pipelines.

See Launch pipelines to start executing workflows in your Azure Batch compute environment.

Managed identity

To use managed identities, Platform requires requires Nextflow version 24.06.0-edge or later. Add export NXF_VER=24.06.0-edge to the Global Nextflow config field in advanced options for your compute environment to use this Nextflow version by default (see manual instructions above).

Nextflow can authenticate to Azure services using a managed identity. This method offers enhanced security compared to access keys, but must run on Azure infrastructure.

When you use a manually configured compute environment with a managed identity attached to the Azure Batch Pool, Nextflow can use this managed identity for authentication. However, Platform still needs to use access keys or an Entra service principal to submit the initial task to Azure Batch to run Nextflow, which will then proceed with the managed identity for subsequent authentication.

1. In Azure, create a user-assigned managed identity. See Manage user-assigned managed identities for detailed steps. After creation, record the Client ID of the managed identity.
2. The user-assigned managed identity must have the necessary access roles for Nextflow. See Required role assignments for more information.
3. Associate the user-assigned managed identity with the Azure Batch Pool. See Set up managed identity in your Batch pool for more information.
4. When you set up the Platform compute environment, select the Azure Batch pool by name and enter the managed identity client ID in the specified field as instructed above.

When you submit a pipeline to this compute environment, Nextflow will authenticate using the managed identity associated with the Azure Batch node it runs on, rather than relying on access keys.