Changelog

New features​

Resource Labels​

In Nextflow Tower 22.3, Seqera Labs introduced resource labels — a flexible tagging system for the cloud services consumed by a run. Workspace administrators can now customize the resource labels associated with pipelines, actions, and runs. This improves the feature’s flexibility as resource labels are no longer inherited only from the compute environment.

In Tower Enterprise 22.4, an administrator can now:

• Override and save the resource labels automatically assigned to a pipeline.
• The pipeline will have a different resource label set from its associated compute environment. Resource labels added to the pipeline propagate to the cloud provider, without being permanently associated with the compute environment in Tower.
• If a maintainer edits a pipeline and changes the compute environment, the resource labels field is updated with the resource labels of the new compute environment.
• Override and save the resource labels associated with an action, following the same logic as pipelines above.
• Override the resource labels associated with a workflow run before launch, enabling job-level tagging.
• The resource labels tied to a workflow run are associated with specific cloud resources that do not include all resources tagged when a compute environment is created.

All runs view​

A comprehensive new view of All runs accessible to each user across the entire Tower instance is now available. This feature is especially useful for monitoring multiple workspaces at once and identifying execution patterns across workspaces and organizations.

Segmented by organizations and workspaces, the interface facilitates overall status monitoring and early detection of execution issues, such as pipeline-related problems or infrastructure issues that can affect multiple workspaces simultaneously.

The All runs view is accessible via the user menu.

Wave support for Tower Enterprise​

All Tower instances with internet access can now connect to the Seqera Labs Wave container service to leverage its container augmentation and Fusion v2 file system capabilities. See the Wave containers documentation for more information about Wave containers.

The Wave integration also allows for the secure transfer of credentials required to access private registries between services. See the Tower documentation to learn how to use the feature in your enterprise installation.

Fusion file system support​

Tower 22.4 adds official support for the Fusion file system. Fusion file system is a lightweight client that enables containerized tasks to access data in Amazon S3 (and other object stores in future) using POSIX file access semantics. Depending on your data handling requirements, Fusion 2.0 improves pipeline throughput and/or reduces cloud computing costs. For additional information on Fusion 2.0 and newly published benchmark results, see the recent article Breakthrough performance and cost-efficiency with the new Fusion file system. The Wave service is a prerequisite for using the Fusion file system.

Resuming runs on a different compute environment​

Tower 22.4 allows users with sufficient permissions to change their compute environment when resuming a run. Users with a maintainer role or above can now select a new compute environment when resuming a run.

This is especially useful if the original run failed due to infrastructure limitations of the compute environment, such as insufficient memory being available to a task. Now, it is possible to select a new compute environment when the run is resumed, without the need to restart from the first task.

The only requirement is that the new compute environment has access to the original run workdir.

Other improvements​

• Update to Java 17
• Support for Gitea credentials and repositories
• UI fixes in the run detail page
  • Alphabetical sorting for reports
  • Horizontal scrolling for log window
• ECS configuration in the advanced setup for AWS compute environments
• Nextflow: Support for S3 Glacier file retrieval
• Nextflow: Define the storage class for published files
• Actions: duplicate the launch for every run from an action to ease management and retrieval (this change is not retroactive — old actions’ runs need to be relaunched for changes to take effect)

Breaking changes and warnings​

Warnings​

1. The default nf-launcher image includes a curl command which will fail if your Tower is secured with a private TLS certificate. To mitigate this problem, please see these instructions.
2. This version requires all database connections to use the following configuration value: TOWER_DB_DRIVER=org.mariadb.jdbc.Driver. Please update your configuration if you are upgrading. All other database configuration values should remain unchanged.
3. This version expects the use of HTTPS by default for all browser client connections. If your Tower installation requires the use of unsecured HTTP, set the following environment variable in the infrastructure hosting the Tower application: TOWER_ENABLE_UNSAFE_MODE=true.
4. If you're upgrading from a version of Tower prior to 21.04.x, please update your implementation to 21.04.x before installing this release.

Upgrade steps​

This Tower version requires a database schema update. Follow these steps to update your DB instance and the Tower installation.

!!! warning To ensure no data loss, the database volume must be persistent on the local machine. Use the volumes key in the db or redis section of your docker-compose.yml file to specify a local path to the DB or Redis instance.

1. Make a backup of the Tower database.

2. Download and update your container versions.

3. Redeploy the Tower application:

   docker compose:

   • To migrate the database schema, restart the application with docker compose down, then docker compose up.

   kubernetes:

   • Update the cron service with kubectl apply -f tower-cron.yml. This will automatically migrate the database schema.
   • Update the frontend and backend services with kubectl apply -f tower-srv.yml.

   custom deployment:

   • Run the /migrate-db.sh script provided in the backend container. This will migrate the database schema.
   • Deploy Tower following your usual procedures.

Nextflow launcher image​

If you must host your nf-launcher container image on a private image registry, copy the nf-launcher image to your private registry. Then update your tower.env with the following environment variable:

TOWER_LAUNCH_CONTAINER=<FULL_PATH_TO_YOUR_PRIVATE_IMAGE>

!!! warning If you're using AWS Batch, you will need to configure a custom job definition and populate the TOWER_LAUNCH_CONTAINER with the job definition name instead.

Sharing feedback​

Share your feedback via support.seqera.io.

New features​

Resource labels​

Tower now supports applying resource labels to compute environments and other Tower elements. This offers a flexible tagging system for annotation and tracking of the cloud services consumed by a run.

Resource labels are sent to the service provider for each cloud compute environment in key=value format. They can be created, edited, and applied by a workspace admin or owner.

Note: Resource labels modified on your cloud provider platform do not update in Tower automatically.

Dashboard​

Tower 22.3 introduces a dashboard interface to view total runs and run status, filtered by organization or user workspace. This facilitates overall run status monitoring and early detection of execution issues.

Select Dashboard from the user menu in the Tower web UI.

Google Batch support​

Tower now supports Google Cloud Batch compute environments. Google Cloud Batch is a comprehensive cloud service suitable for multiple use cases, including HPC, AI/ML, and data processing. Tower now provides an integration with your existing Google Cloud account via the Batch API. While it is similar to the Google Cloud Life Sciences API, Google Cloud Batch offers a broader set of capabilities.

Google Cloud Batch automatically provisions resources, manages capacity, and allows batch workloads to run at scale. The API has built-in support for data ingestion from Google Cloud Storage buckets. This makes data ingestion and sharing datasets efficient and reliable.

This is a Beta Tower feature — more capability will be added as Nextflow Google Cloud Batch support evolves.

Admin panel enhancements​

The Tower admin panel now provides additional user and organization management features.

• From the Users tab, admins can view all users, assign or remove users, and change user roles within an organization.
• From the Organizations tab, admins can view organizations, assign or remove users, and manage the user roles within an organization.

Resource optimization (technology preview)​

Tower Cloud now supports cloud resource optimization when running pipelines. Using the extensive resource usage data which Tower already collects for each pipeline run, a set of per-process resource recommendations is generated and can be applied to subsequent runs. This feature is geared to optimize resource use significantly, while being conservative enough to ensure that pipelines run reliably.

This feature is currently only available on Tower Cloud (tower.nf). For more information about this optional feature, contact us.

Wave containers (technology preview)​

Tower now supports the Nextflow Wave container provisioning and augmentation service. When a pipeline is run in Nextflow using Wave, the Wave service uses a Dockerfile stored in the process directory to build a container in the target registry. When the container to be used for process execution is returned, the Wave service can add functional layers and data on-the-fly before it is returned to Nextflow for actual process execution.

Wave also enables the use of private container registries in Nextflow — registry credentials stored in Tower are used to authenticate to private container registries with the Wave service.

The Wave container provisioning service is available free of charge as a technology preview to all Nextflow and Tower users. During the preview period, anonymous users can build up to 10 container images per day and pull 100 containers per hour. Tower authenticated users can build 100 container images per hour and pull 1000 containers per minute. After the preview period, we plan to make the Wave service available free of charge to academic users and open-source software (OSS) projects.

See here for an introductory overview of Wave containers on the Nextflow blog, and here for a live demo and introduction of Wave from the Nextflow 2022 Summit, by Seqera Labs co-founder and CTO Paolo di Tommaso.

This feature is currently only available on Tower Cloud (tower.nf). For more information about this optional feature, contact us.

Fusion file system (technology preview)​

Fusion is a virtual distributed file system which allows data hosted in AWS S3 buckets to be accessed directly by the file system interface used by pipeline tools. This means that Nextflow pipelines can use an S3 bucket as the work directory and pipeline tasks can access the S3 bucket natively as a local file system path.

Fusion, as used by the Wave container provisioning service, is available free of charge as a technology preview to all Nextflow and Tower users. After the preview period, we plan to make the service available free of charge to academic users and open-source software (OSS) projects.

This feature is currently only available on Tower Cloud (tower.nf). For more information about this optional feature, contact us.

Other improvements​

• Owners have full permissions for all workspaces in their organization.
• Navigation restyling.
• Launch/relaunch form allows head node resource customization.
• Runs page supports task name search in Tasks table.
• Expand boot EBS volume size.
• Label and resource label APIs are now exposed.
• The amount of usable datasets (and dataset versions) per organization has been raised to 100 records by default.
• Customize head node resources from the launch/relaunch form.
  • As a user (with maintainer permissions) it is now possible to launch a pipeline in a Tower cloud environment, specifying the head node resources (memory and CPU) from the launch form. This allows you to properly dimension resources and avoid pipeline crashes. This feature is available for AWS, Google Life Sciences, and Kubernetes Compute Environments.
• The Revision field in the launch form has been extended to allow a maximum length of 100 characters.
• Improve SSH connector resilience + UGE qstat.

Fixes​

• BitBucketServer Git provider.
• Container registry name.
• Missing file existence check for Google Life Sciences.
• Resume functionality on Google Life Sciences.
• Improved error traceability when an exception occurs in the prerun script block.
• Fixed a bug that prevented a run to be resumed for users with launch permissions.
• Saving status for a job fails when a DB exception occurs.
• Escape qstat command for Altair PBS batch scheduler.

Breaking changes and warnings​

Breaking changes​

• In previous versions, some assets required by Batch Forge were downloaded from S3 bucket named nf-xpack.s3.eu-west-1.amazonaws.com. As of version 22.3.x, those assets are now downloaded from [https://nf-xpack.seqera.io](https://nf-xpack.seqera.io/). Make sure your network policy allows access to the seqera.io domain.

• Use of the resource labels feature with AWS Batch requires an update of the IAM policy used by the account running Tower. The required changes can be found here.

• In previous versions, if Tower was configured to authenticate to AWS via instance role, Batch Forge would assign this same IAM Role as the Head Job role and Compute Job role of the AWS Batch compute environment it created. As of version 22.3.1, you must explicitly assign these job roles during the AWs Batch compute environment creation process.

Warnings​

1. This version requires all database connections to use the following configuration value: TOWER_DB_DRIVER=org.mariadb.jdbc.Driver. Please update your configuration if you are upgrading. All other database configuration values should remain unchanged.
2. This version expects the use of HTTPS by default for all browser client connections. If your Tower installation requires the use of unsecured HTTP, set the following environment variable in the infrastructure hosting the Tower application: TOWER_ENABLE_UNSAFE_MODE=true.
3. If you are upgrading from a version of Tower prior to 21.04.x, please update your implementation to 21.04.x before installing this release.

Database schema​

This Tower version requires a database schema update. Follow these steps to update your DB instance and the Tower installation.

1. Make a backup of the Tower database.

2. Download and update your container versions.

3. Redeploy the Tower application:

   docker compose:

   • Restart the application with docker compose restart. This will automatically migrate the database schema.

   kubernetes:

   • Update the cron service with kubectl apply -f tower-cron.yml. This will automatically migrate the database schema.
   • Update the frontend and backend services with kubectl apply -f tower-srv.yml.

   custom deployment:

   • Run the /migrate-db.sh script provided in the backend container. This will migrate the database schema.
   • Deploy Tower following your usual procedures.

Nextflow launcher image​

If you must host your nf-launcher container image on a private image registry, copy the nf-launcher image to your private registry. Then update your tower.env with the following environment variable:

TOWER_LAUNCH_CONTAINER=<FULL_PATH_TO_YOUR_PRIVATE_IMAGE>

!!! warning If you're using AWS Batch, you will need to configure a custom job-definition and populate the TOWER_LAUNCH_CONTAINER with the job-definition name instead.

New compute environments​

New compute environment options are available:

• googlebatch-platform: Google Batch cloud compute service

Sharing feedback​

You can share your feedback via https://support.seqera.io.

Changelog​

• Add support for Service Account, VPC and Subnetwork in Google Batch Advanced Settings.

• Add support for AWS Batch SPOT_PRICE_CAPACITY_OPTIMIZED allocation strategy.

• Add page size selector and pagination for the tasks table in the workflow details page.

• Add support for Fusion to EKS and GKE platform providers.

• Add support for Secrets for Google Batch and Google LS.

• Improve responsiveness for the workflow runs list page.

• Add support for downloading and previewing bucket files through Data Explorer.

• Adds the possibility to specify a custom base href (useful in a reverse proxy scenario).

• Fix disabled search bar after getting 0 results for a search in the workflow reports tab.

• Add download as json option for workflow run parameters.

• Decrease audit log lifespan to 90 days.

• Add support for uploading files through Data Explorer.

• Add support for Nextflow cloudcache plugin.

• Add support for navigating workflow and task work directories using Data Explorer.

• Apply new branding to UI and copy.

• Bump avatar file size limit to 200KB.

• Improve auto-suggested datalink name.

• Disable upload functionality on public Data Links.

• Fix tasks total number getting stuck after filtering.

• Previewing text files in Data Explorer now capped at 2000 lines to prevent browser from hanging.

• Enable instance types selection for dragen queue.

• Fix display of error messages in pipeline input form.

• Fix report preview dialog height.

• Adds a new attempt column to the task table.

• Deprecate Fusion V1.

• Add support for selecting pipeline input values using Data Explorer.

• Add a per workspace and global feature toggle for Data Explorer.

• Update Azure icon (Azure rebranding from May 2021).

• Add support for custom network and subnetwork to Google Cloud Batch compute enviroment.

• Bump nf-launcher:j17-23.04.3