Google Cloud Batch

This guide assumes you have an existing Google Cloud account. Sign up for a free account here. Seqera Platform provides integration to Google Cloud via the Batch API.

The guide is split into two parts:

1. How to configure your Google Cloud account to use the Batch API.
2. How to create a Google Cloud Batch compute environment in Seqera.

Configure Google Cloud

Create a project

Go to the Google Project Selector page and select an existing project, or select Create project.

Enter a name for your new project, e.g., tower-nf.

If you are part of an organization, the location will default to your organization.

Enable billing

See here to enable billing in your Google Cloud account.

Enable APIs

See here to enable the following APIs for your project:

• Batch API
• Compute Engine API
• Cloud Storage API

Select your project from the dropdown menu and select Enable.

Alternatively, you can enable each API manually by selecting your project in the navigation bar and visiting each API page:

• Batch API

• Compute Engine API

• Cloud Storage API

IAM

Seqera requires a service account with appropriate permissions to interact with your Google Cloud resources. As an IAM user, you must have access to the service account that submits Batch jobs.

By default, Google Cloud Batch uses the default Compute Engine service account to submit jobs. This service account is granted the Editor (roles/Editor) role. While this service account has the necessary permissions needed by Seqera, this role is not recommended for production environments. Control job access using a custom service account with only the permissions necessary for Seqera to execute Batch jobs instead.

Service account permissions

Create a custom service account with at least the following permissions:

• Batch Agent Reporter (roles/batch.agentReporter) on the project
• Batch Job Editor (roles/batch.jobsEditor) on the project
• Logs Writer (roles/logging.logWriter) on the project (to let jobs generate logs in Cloud Logging)
• Service Account User (roles/iam.serviceAccountUser)

If your Google Cloud project does not require access restrictions on any of its Cloud Storage buckets, you can grant project Storage Admin (roles/storage.admin) permissions to your service account to simplify setup. To grant access only to specific buckets, add the service account as a principal on each bucket individually. See Cloud Storage bucket below.

User permissions

Ask your Google Cloud administrator to grant you the following IAM user permissions to interact with your custom service account:

• Batch Job Editor (roles/batch.jobsEditor) on the project
• Service Account User (roles/iam.serviceAccountUser) on the job's service account (default: Compute Engine service account)
• View Service Accounts (roles/iam.serviceAccountViewer) on the project

To configure a credential in Seqera, you must first create a service account JSON key file:

1. In the Google Cloud navigation menu, select IAM & Admin > Service Accounts.

2. Select the email address of the service account.

   The Compute Engine default service account is not recommended for production environments due to its powerful permissions. To use a service account other than the Compute Engine default, specify the service account email address under Advanced options on the Seqera compute environment creation form.

3. Select Keys > Add key > Create new key.

4. Select JSON as the key type.

5. Select Create.

A JSON file is downloaded to your computer. This file contains the credential needed to configure the compute environment in Seqera.

You can manage your key from the Service Accounts page.

Cloud Storage bucket

Google Cloud Storage is a type of object storage. To access files and store the results for your pipelines, create a Cloud bucket that your Seqera service account can access.

Create a Cloud Storage bucket

1. In the hamburger menu (≡), select Cloud Storage.

2. From the Buckets tab, select Create.

3. Enter a name for your bucket. You will reference this name when you create the compute environment in Seqera.

4. Select Region for the Location type and select the Location for your bucket. You'll reference this location when you create the compute environment in Seqera.

   The Batch API is available in a limited number of locations. These locations are only used to store metadata about the pipeline operations. The storage bucket and compute resources can be in any region.

5. Select Standard for the default storage class.

6. To restrict public access to your bucket data, select the Enforce public access prevention on this bucket checkbox.

7. Under Access control, select Uniform.

8. Select any additional object data protection tools, per your organization's data protection requirements.

9. Select Create.

Assign bucket permissions

1. After the bucket is created, you are redirected to the Bucket details page.
2. Select Permissions, then Grant access under View by principals.
3. Copy the email address of your service account into New principals.
4. Select the Storage Admin role, then select Save.

You've created a project, enabled the necessary Google APIs, created a bucket, and created a service account JSON key file with the required credentials. You now have what you need to set up a new compute environment in Seqera.

Seqera compute environment

Your Seqera compute environment uses resources that you may be charged for in your Google Cloud account. See Cloud costs for guidelines to manage cloud resources effectively and prevent unexpected costs.

After your Google Cloud resources have been created, create a new Seqera compute environment:

1. In a workspace, select Compute Environments > New Environment.
2. Enter a descriptive name for this environment, e.g., Google Cloud Batch (europe-north1).
3. Select Google Cloud Batch as the target platform.

Credentials

1. From the Credentials drop-down, select existing Google credentials or select + to add new credentials. If you choose to use existing credentials, skip to the next section.
2. Enter a name for the credentials, e.g., Google Cloud Credentials.
3. Paste the contents of the JSON file created previously in the Service account key field.

Location and work directory

Select the Location where you will execute your pipelines. See Location to learn more.

In the Pipeline work directory field, enter your storage bucket URL, e.g., gs://my-bucket. This bucket must be accessible in the location selected in the previous step.

When you specify a Cloud Storage bucket as your work directory, this bucket is used for the Nextflow cloud cache by default. You can specify an alternative cache location with the Nextflow config file field on the pipeline launch form.

Seqera features

Select Enable Wave containers to facilitate access to private container repositories and provision containers in your pipelines using the Wave containers service. See Wave containers for more information.

Select Enable Fusion v2 to allow access to your Google Cloud Storage data via the Fusion v2 virtual distributed file system. This speeds up most data operations. The Fusion v2 file system requires Wave containers to be enabled. See Fusion file system for configuration details.

Use Fusion v2

The compute recommendations below are based on internal benchmarking performed by Seqera. Benchmark runs of nf-core/rnaseq used profile test_full, consisting of an input dataset with 16 FASTQ files and a total size of approximately 123.5 GB.

1. Use Seqera Platform version 23.1 or later.
2. Use a Google Cloud Storage bucket as the pipeline work directory.
3. Enable Wave containers and Fusion v2.
4. Specify suitable virtual machine types and local storage settings, or accept the default machine settings listed below. An n2-highmem-16-lssd VM or larger is recommended for production use.

To specify virtual machine settings in Platform during compute environment creation, use the Global Nextflow config field to apply custom Nextflow process directives to all pipeline runs launched with this compute environment.

To specify virtual machine settings per pipeline run in Platform, or as a persistent configuration in your Nextflow pipeline repository, use Nextflow process directives. See Google Cloud Batch process definition for more information.

When Fusion v2 is enabled, the following virtual machine settings are applied:

• A 375 GB local NVMe SSD is selected for all compute jobs.
• If you do not specify a machine type, a VM from families that support local SSDs is selected.
• Any machine types you specify in the Nextflow config must support local SSDs.
• Local SSDs are only offered in multiples of 375 GB. You can increment the number of SSDs used per process with the disk directive to request multiples of 375 GB. To work with files larger than 100 GB, use at least two SSDs (750 GB or more).
• Fusion v2 can also use persistent disks for caching. Override the disk requested by Fusion using the disk directive and the type: pd-standard.
• The machineType directive can be used to specify a VM instance type, family, or custom machine type in a comma-separated list of patterns. For example, c2-*, n1-standard-1, custom-2-4, n*, m?-standard-*.

Wave containers and Fusion v2 are recommended features for added capability and improved performance, but neither are required to execute workflows in your compute environment.

GCP resources

Enable Spot to use Spot instances, which have significantly reduced cost compared to on-demand instances.

Apply Resource labels to the cloud resources consumed by this compute environment. Workspace default resource labels are prefilled.

Scripting and environment variables

• Expand Staging options to include:

  • Optional pre- or post-run Bash scripts that execute before or after the Nextflow pipeline execution in your environment.
  • Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the Nextflow config file field in the pipeline launch form. These values can be overridden during pipeline launch.

  Configuration settings in this field override the same values in the pipeline repository nextflow.config file. See Nextflow config file for more information on configuration priority.

• Specify custom Environment variables for the head and compute jobs.

Advanced options

If you use VM instance templates for the head or compute jobs (see step 6 below), resource allocation and networking values specified in the templates override any conflicting values you specify while creating your Seqera compute environment.

1. Enable Use Private Address to ensure that your Google Cloud VMs aren't accessible to the public internet.

2. Use Boot disk size to control the persistent disk size that each task and the head job are provided.

3. Use Head Job CPUs and Head Job Memory to specify the CPUs and memory allocated for the head job.

4. Use Service Account email to specify a service account email address other than the Compute Engine default to execute workflows with this compute environment (recommended for productions environments).

5. Use VPC and Subnet to specify the name of a VPC network and subnet to be used by this compute environment. If your organization's VPC architecture relies on network tags, you can apply network tags to VM instance templates used for the Nextflow head and compute jobs (see below).

   You must specify both a VPC and Subnet for your compute environment to use either.

6. Use Head job instance template and Compute jobs instance template to specify the name or fully-qualified reference of a VM instance template, without the template:// prefix, to use for the head and compute jobs. VM instance templates allow you to define the resources allocated to Batch jobs. Configuration values defined in a VM instance template override any conflicting values you specify while creating your Seqera compute environment.

   You can use network tags in VM instance templates to enable cross-network and cross-project distribution of compute resources. This is useful if your head and compute instances must reside in different GCP projects or across isolated networking infrastructures. Note that the use of network tags does not affect the resource labels applied to your compute environment.

   Seqera does not validate the VM instance template you specify in these fields. Generally, use templates that define only the machine type, network, disk, and configuration values that will not change across multiple VM instances and Seqera compute environments. See Create instance templates for instructions to create your instance templates.

   To prevent errors during workflow execution, ensure that the instance templates you use are suitably configured for your needs with an appropriate machine type. You can define multiple instance templates with varying machine type sizes in your Nextflow configuration using the machineType process directive (e.g., process.machineType = 'template://my-template-name'). You can use process selectors to assign separate templates to each of your processes.

Select Create to finalize the compute environment setup.

See Launch pipelines to start executing workflows in your Google Cloud Batch compute environment.