Azure Batch

This guide assumes you already have an Azure account with a valid Azure Subscription. For details, visit Azure Free Account. Ensure you have sufficient permissions to create resource groups, an Azure Storage account, and a Batch account.

Concepts

Accounts

Seqera Platform relies on an existing Azure Storage and Azure Batch account. You need at least 1 valid Azure Storage account and Azure Batch account within your subscription.

Azure uses 'accounts' for each service. For example, an Azure Storage account will house a collection of blob containers, file shares, queues, and tables. While you can have multiple Azure Storage and Azure Batch accounts in an Azure subscription, each compute environment on the platform can only use one of each (one storage and one Batch account). You can set up multiple compute environments on the platform with different credentials, storage accounts, and Batch accounts.

Resource group

To create Azure Batch and Azure Storage accounts, first create a resource group in your preferred region.

A resource group can be created while creating an Azure Storage Account or Azure Batch account.

Regions

Azure resources can operate across regions, but this incurs additional costs and security requirements. It is recommended to place all resources in the same region. See the Azure product page on data residency for more information.

Resource group

A resource group in Azure is a unit of related resources in Azure. As a rule of thumb, resources that have a similar lifecycle should be within the same resource group. You can delete a resource group and all associated components together. We recommend placing all platform compute resources in the same resource group, but this is not necessary.

Create a resource group

1. Log in to your Azure account, go to the Create Resource group page, and select Create new resource group.
2. Enter a name for the resource group, e.g., towerrg.
3. Choose the preferred region.
4. Select Review and Create to proceed.
5. Select Create.

Storage account

After creating a resource group, set up an Azure storage account.

Create a storage account

1. Log in to your Azure account, go to the Create storage account page, and select Create a storage account.

   If you haven't created a resource group, you can do so now.

2. Enter a name for the storage account (e.g., towerrgstorage).

3. Choose the preferred region (same as the Batch account).

4. The platform supports any performance or redundancy settings — select the most appropriate settings for your use case.

5. Select Next: Advanced.

6. Enable storage account key access.

7. Select Next: Networking.

   • Enable public access from all networks. You can enable public access from selected virtual networks and IP addresses, but you will be unable to use Forge to create compute resources. Disabling public access is not supported.

8. Select Data protection.

   • Configure appropriate settings. All settings are supported by the platform.

9. Select Encryption.

   • Only Microsoft-managed keys (MMK) are supported.

10. In tags, add any required tags for the storage account.

11. Select Review and Create.

12. Select Create to create the Azure Storage account.

13. You will need at least one blob storage container to act as a working directory for Nextflow.

14. Go to your new storage account and select + Container to create a new Blob storage container. A new container dialogue will open. Enter a suitable name, e.g., towerrgstorage-container.

15. Go to the Access Keys section of your new storage account (towerrgstorage in this example).

16. Store the access keys for your Azure Storage account, to be used when you create a Seqera compute environment.

Blob container storage credentials are associated with the Batch pool configuration. Avoid changing these credentials in your Seqera instance after you have created the compute environment.

Batch account

After you have created a resource group and storage account, create a Batch account.

Create a Batch account

1. Log in to your Azure account and select Create a batch account on this page.

2. Select the existing resource group or create a new one.

3. Enter a name for the Batch account, e.g., towerrgbatch.

4. Choose the preferred region (same as the storage account).

5. Select Advanced.

6. For Pool allocation mode, select Batch service.

7. For Authentication mode, ensure Shared Key is selected.

8. Select Networking. Ensure networking access is sufficient for the platform and any additional required resources.

9. In tags, add any required tags for the Batch account.

10. Select Review and Create.

11. Select Create.

12. Go to your new Batch account, then select Access Keys.

13. Store the access keys for your Azure Batch account, to be used when you create a Seqera compute environment.

    A newly-created Azure Batch account may not be entitled to create virtual machines without making a service request to Azure. See Azure Batch service quotas and limits for more information.

14. Select the + Quotas tab of the Azure Batch account to check and increase existing quotas if necessary.

15. Select + Request quota increase and add the quantity of resources you require. Here is a brief guideline:

    • Active jobs and schedules: Each Nextflow process will require an active Azure Batch job per pipeline while running, so increase this number to a high level. See here to learn more about jobs in Azure Batch.
    • Pools: Each platform compute environment requires one Azure Batch pool. Each pool is composed of multiple machines of one virtual machine size.

    To use separate pools for head and compute nodes, see this FAQ entry.

    • Batch accounts per region per subscription: Set this to the number of Azure Batch accounts per region per subscription. Only one is required.
    • Spot/low-priority vCPUs: Platform does not support spot or low-priority machines when using Forge, so when using Forge this number can be zero. When manually setting up a pool, select an appropriate number of concurrent vCPUs here.
    • Total Dedicated vCPUs per VM series: See the Azure documentation for virtual machine sizes to help determine the machine size you need. We recommend the latest version of the ED series available in your region as a cost-effective and appropriately-sized machine for running Nextflow. However, you will need to select alternative machine series that have additional requirements, such as those with additional GPUs or faster storage. Increase the quota by the number of required concurrent CPUs. In Azure, machines are charged per cpu minute so there is no additional cost for a higher number.

Compute environment

There are two ways to create an Azure Batch compute environment in Seqera Platform:

• Batch Forge: Automatically creates Azure Batch resources.
• Manual: For using existing Azure Batch resources.

Batch Forge

Batch Forge automatically creates resources that you may be charged for in your Azure account. See Cloud costs for guidelines to manage cloud resources effectively and prevent unexpected costs.

Create a Batch Forge Azure Batch compute environment:

1. In a workspace, select Compute Environments > New Environment.

2. Enter a descriptive name, e.g., Azure Batch (east-us).

3. Select Azure Batch as the target platform.

4. Choose existing Azure credentials or add a new credential. If you are using existing credentials, skip to step 7.

   You can create multiple credentials in your Seqera environment.

5. Enter a name for the credentials, e.g., Azure Credentials.

6. Add the Batch account and Blob Storage account names and access keys.

7. Select a Region, e.g., eastus.

8. In the Pipeline work directory field, enter the Azure blob container created previously, e.g., az://towerrgstorage-container/work.

   When you specify a Blob Storage bucket as your work directory, this bucket is used for the Nextflow cloud cache by default. You can specify an alternative cache location with the Nextflow config file field on the pipeline launch form.

9. Select Enable Wave containers to facilitate access to private container repositories and provision containers in your pipelines using the Wave containers service. See Wave containers for more information.

10. Select Enable Fusion v2 to allow access to your Azure Blob Storage data via the Fusion v2 virtual distributed file system. This speeds up most data operations. The Fusion v2 file system requires Wave containers to be enabled. See Fusion file system for configuration details.

11. Set the Config mode to Batch Forge.

12. Enter the default VMs type, depending on your quota limits set previously. The default is Standard_D4_v3.

13. Enter the VMs count. If autoscaling is enabled (default), this is the maximum number of VMs you wish the pool to scale up to. If autoscaling is disabled, this is the fixed number of virtual machines in the pool.

14. Enable Autoscale to scale up and down automatically, based on the number of pipeline tasks. The number of VMs will vary from 0 to VMs count.

15. Enable Dispose resources for Seqera to automatically delete the Batch pool if the compute environment is deleted on the platform.

16. Select or create Container registry credentials to authenticate a registry (used by the Wave containers service). It is recommended to use an Azure Container registry within the same region for maximum performance.

17. Apply Resource labels. This will populate the Metadata fields of the Azure Batch pool.

18. Expand Staging options to include optional pre- or post-run Bash scripts that execute before or after the Nextflow pipeline execution in your environment.

19. Specify custom Environment variables for the Head job and/or Compute jobs.

20. Configure any advanced options you need:

    • Use Jobs cleanup policy to control how Nextflow process jobs are deleted on completion. Active jobs consume the quota of the Azure Batch account. By default, jobs are terminated by Nextflow and removed from the quota when all tasks succesfully complete. If set to Always, all jobs are deleted by Nextflow after pipeline completion. If set to Never, jobs are never deleted. If set to On success, successful tasks are removed but failed tasks will be left for debugging purposes.
    • Use Token duration to control the duration of the SAS token generated by Nextflow. This must be as long as the longest period of time the pipeline will run.

21. Select Add to finalize the compute environment setup. It will take a few seconds for all the resources to be created before the compute environment is ready to launch pipelines.

See Launch pipelines to start executing workflows in your Azure Batch compute environment.

Manual

This section is for users with a pre-configured Azure Batch pool. This requires an existing Azure Batch account with an existing pool.

Your Seqera compute environment uses resources that you may be charged for in your Azure account. See Cloud costs for guidelines to manage cloud resources effectively and prevent unexpected costs.

Create a manual Seqera Azure Batch compute environment

1. In a workspace, select Compute Environments > New Environment.

2. Enter a descriptive name for this environment, e.g., Azure Batch (east-us).

3. Select Azure Batch as the target platform.

4. Select your existing Azure credentials or select + to add new credentials. If you choose to use existing credentials, skip to step 7.

   You can create multiple credentials in your Seqera environment.

5. Enter a name, e.g., Azure Credentials.

6. Add the Batch account and Blob Storage credentials you created previously.

7. Select a Region, e.g., eastus (East US).

8. In the Pipeline work directory field, add the Azure blob container created previously, e.g., az://towerrgstorage-container/work.

   When you specify a Blob Storage bucket as your work directory, this bucket is used for the Nextflow cloud cache by default. You can specify an alternative cache location with the Nextflow config file field on the pipeline launch form.

9. Set the Config mode to Manual.

10. Enter the Compute Pool name. This is the name of the Azure Batch pool you created previously in the Azure Batch account.

    The default Azure Batch implementation uses a single pool for head and compute nodes. To use separate pools for head and compute nodes (e.g., to use low-priority VMs for compute jobs), see this FAQ entry.

11. Specify custom Environment variables for the Head job and/or Compute jobs.

12. Configure any advanced options you need:

    • Use Jobs cleanup policy to control how Nextflow process jobs are deleted on completion. Active jobs consume the quota of the Azure Batch account. By default, jobs are terminated by Nextflow and removed from the quota when all tasks succesfully complete. If set to Always, all jobs are deleted by Nextflow after pipeline completion. If set to Never, jobs are never deleted. If set to On success, successful tasks are removed but failed tasks will be left for debugging purposes.
    • Use Token duration to control the duration of the SAS token generated by Nextflow. This must be as long as the longest period of time the pipeline will run.

13. Select Add to finalize the compute environment setup. It will take a few seconds for all the resources to be created before you are ready to launch pipelines.

See Launch pipelines to start executing workflows in your Azure Batch compute environment.