Skip to main content

Google Cloud Storage

Nextflow supports Google Cloud Storage. You can access files stored in a Cloud Storage bucket in your pipeline script like any other file in the local file system.

Cloud Storage paths​

To access a Cloud Storage file, prefix the file path with the gs:// scheme and the name of the bucket where it is stored.

For example, to access the file /data/sequences.fa stored in a bucket named my-bucket, use the following fully qualified path:

gs://my-bucket/data/sequences.fa

You can apply the usual file operations to a path handle with the above notation. For example, to print the content of a Cloud Storage file:

println file('gs://my-bucket/data/sequences.fa').text

See Working with files and the Path reference to learn more about available file operations.

Authentication​

Nextflow reads credentials for Google Cloud services, such as Cloud Storage and Cloud Batch, from your environment using Application Default Credentials. Application Default Credentials are designed to use the credentials most natural to the environment in which a tool runs.

The most common case is to use your end-user Google credentials from your workstation. To create them, run the following command and complete the authentication flow:

gcloud auth application-default login

This command writes a credential file to your gcloud configuration directory. Any tool you run on your workstation that reads default credentials uses this file.

The next most common case is running on a Compute Engine VM. In this case, Application Default Credentials uses the Compute Engine Service Account credentials for that VM.

You can also use the GOOGLE_APPLICATION_CREDENTIALS environment variable to specify the location of a service account key file. To create a key file, see Create and delete service account keys in the Google Cloud documentation. Then set the following variable, replacing the path in the example with the path of your key file:

export GOOGLE_APPLICATION_CREDENTIALS="/path/your/file/creds.json"

When using a key file, Nextflow uses the project ID in the key file as the default value of google.project. If google.project is also specified in the Nextflow configuration, it must match the project ID in the key file.

See the Application Default Credentials documentation for how to enable other use cases.

Required permissions​

The identity associated with your credentials must be able to read and write objects in the buckets used by the pipeline, such as the work directory, input data, and published outputs. The Storage Object User role (roles/storage.objectUser) grants the required permissions. For buckets that are only read by the pipeline, the Storage Object Viewer role (roles/storage.objectViewer) is sufficient.

See Google Cloud Batch for the additional roles required by the Google Batch executor.

Requester pays buckets​

To access data in requester pays buckets, specify the project to be billed and enable requester pays in the Nextflow configuration:

google {
    project = '<project-id>'
    enableRequesterPaysBuckets = true
}

Advanced configuration​

See the google config scope for all available options.

On this Page