Seqera Enterprise v23.4.5
Feature updates and improvements
- The standard email login can be disabled via
tower.yml
or an environment variable, provided an alternative OIDC provider is set up first.
tower.yml
or an environment variable, provided an alternative OIDC provider is set up first.Seqera Platform Enterprise version 24.1 introduces three new features: Data Studios (in public preview), Data Explorer, and managed identities. A number of bug fixes and performance enhancements are included in this major release.
Data Studios closes the loop from development to deployment and insights, allowing you to create, manage, and share notebook environments in Seqera with the click of a button. Data Studios makes it seamless to work across teams with multi-user support, built-in authentication, and automatic snapshots as you work.
Data Studios is available as a public preview. Contact support to request access to this feature.
Data Explorer allows you to browse and interact with remote data repositories from organization workspaces in Seqera Platform. It supports AWS S3, Azure Blob Storage, and Google Cloud Storage repositories.
Data Explorer is now Generally Available (GA) and supports multi-file and multi-folder download. This allows users to download entire datasets of pipeline results to share with their team.
User roles provide flexibility for admins to provide users with the permissions they need, without compromising security. The Connect user role has been added to the existing user roles with functionality related to Data Studios:
See User roles for more information.
Managed identities offer significant advantages for High-Performance Computing (HPC) environments by enabling granular access control for individual workspace users. Unlike traditional SSH credentials that grant all workspace users access to HPC clusters with the same service account, managed identities ensure each user’s activity is logged under their own credentials.
See Managed identities for more information.
tower.yml
or an environment variable, provided an alternative OIDC provider is set up first. See Basic configuration for more information.MariaDB driver: New MySQL connection parameter required
MariaDB driver 3.x requires a special parameter in the connection URL to connect to a MySQL database:
jdbc:mysql://<domain>:<port>/tower?permitMysqlScheme=true
All deployments using a MySQL database (regardless of version: 5.6, 5.7, or 8) should be updated accordingly when upgrading to Platform version 24.1 or later.
MariaDB driver: No truncation support for MySQL 5.6
The MariaDB driver has dropped support for the jdbcCompliantTruncation
parameter, which was true by default and set the STRICT_TRANS_TABLES
SQL mode. The STRICT_TRANS_TABLES
mode produces an error when the value of a VARCHAR column exceeds its limit, instead of truncating it to fit. Most common installations of MySQL 5.7 and 8 already include this mode at the server level, but the Docker container version of MySQL 5.6 does not.
The SQL mode must be set explicitly through the connection URL for deployments still using MySQL 5.6:
jdbc:mysql://<domain>:<port>/tower?permitMysqlScheme=true&sessionVariables=sql_mode='STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION'
Micronaut property key changes
The property that determines the expiration time of the JWT access token (used for authenticating web sessions and Nextflow-Platform interactions) has changed:
Previous | New |
---|---|
micronaut.security.token.jwt.generator.access-token.expiration | micronaut.security.token.generator.access-token.expiration |
Enterprise deployments that have customized this value previously will need to adopt the new format.
running
state as active run data may be lost.tower.env
, configmap.yml
, etc.):
JAVA_OPTS: -Xms1000M -Xmx2000M -XX:MaxDirectMemorySize=800m -Dio.netty.maxDirectMemory=0 -Djdk.nio.maxCachedBufferSize=262144
Docker Compose deployments require downtime while upgrading services. Restarting the application may take several minutes. See Docker compose deployment for more information.
For Kubernetes deployments, apply the 24.1 tower-cron.yml
to your cron pod and wait for the cron pod to be running before applying the tower-svc.yml
to your backend pod and restarting the service. If the cron pod update is interrupted, you may need to restore the instance from your DB backup and start again. See Kubernetes deployment for more information.
For custom deployments with third-party services such as ArgoCD, contact support for assistance during upgrade.
docker-compose
in deployment files to docker compose
.Seqera Platform Enterprise version 23.4 introduces a redesigned UI, VM instance template support for Google Cloud Batch, and database deployment improvements. A number of bug fixes and performance enhancements have also been included in this major release.
Version 23.4.6 is the baseline for the 23.4 major release cycle.
Seqera Platform 23.4 features refreshed forms and UI elements aimed at enhancing user experience and streamlining form navigation. This redesign encompasses all application interface forms, including pipelines, compute environments, Data Explorer, and administrative pages to create a more intuitive user journey.
Seqera now supports VM instance templates for head and compute jobs in Google Cloud Batch compute environments. VM instance templates provide a convenient way to save a VM configuration, thereby allowing you to define the resources allocated to Batch jobs.
Platform Enterprise 23.4 includes an update to the Enterprise licensing model. While Seqera support will contact affected customers to update licenses, the license manager remains backward compatible with existing licenses. For standard Enterprise licenses, no customer action is required. License limits are enforced remotely — if your Enterprise license includes custom limits, contact Seqera support to ensure a seamless transition.
Seqera Platform Enterprise version 23.4 officially supports MySQL 8.0. The default MySQL version in the docker-compose.yml
and tower-cron.yml
deployment templates for Docker Compose and Kubernetes deployments has been updated from 5.7 to 8.0 in the Seqera version 23.4 documentation. See Upgrade steps below for instructions to update your Seqera databases from older versions to MySQL 8.
Previous versions of the deployment template files are still available in Platform docs versions 23.3 and older.
New migrate-db container for database migration
In version 23.4, database migration logic has moved to a new container separate from the backend cron container. This generates a better separation of responsibility across various components of the Platform infrastructure. The change is trivial for Kubernetes installations. For Docker Compose, the startup lifecycle of the containers is improved, with better dependency handling among them. See Upgrade steps below for more information to update and migrate your Seqera databases.
Docker Compose V2 supersedes standalone docker-compose for Docker installs
The Docker Compose CLI plugin replaces the standalone docker-compose
binary, which was deprecated by DockerHub in July 2023 by Compose V2. The installation documentation now uses the docker compose
subcommand for the Docker CLI when using compose files.
Cloud compute environments use cloud cache by default
When a cloud storage location is provided as the pipeline work directory in a cloud compute environment, a scratch folder is created in that location to be used for the Nextflow process cache by default. This can be overridden with an alternate cache entry in your Nextflow configuration.
Login redirection logic update
Login redirection logic has changed in version 23.4. Seqera now prepends the TOWER_SERVER_URL
(or tower.serverUrl
in tower.yml
configuration) to the authentication redirect URL during the login flow. This is useful when your server URL contains a contextual path.
If you specify a DNS name as your TOWER_SERVER_URL
, but access your Seqera instance using a different address (such as using an IP address that resolves to the server URL asynchronously), user login will not resolve.
Revert default Tower name changes in documentation
A previous iteration of the rebranded Seqera documentation noted seqera
as the default and example value for certain variables (such as default database names). The rebranding from Nextflow Tower to Seqera Platform is an ongoing, incremental process and as such, legacy tower
values and naming conventions used by the Seqera backend will remain in place until a future release. Updates to configuration variables and values will be communicated well in advance to prepare users for any breaking changes.
ARM64 CPU architecture support enabled by default
The Use Graviton CPU architecture option is now available by default during AWS Batch compute environment creation. The TOWER_ENABLE_ARM64
configuration environment variable is no longer needed to enable ARM64 CPU architecture support.
Data Explorer default set to false
In previous versions, Data Explorer was enabled by default using TOWER_DATA_EXPLORER_ENABLED=true
. From version 2.4.3, the default is TOWER_DATA_EXPLORER_ENABLED=false
. If you have upgraded from a previous version and no longer have access to Data Explorer, please check and update your environment variables accordingly.
This version requires a database schema update. Follow these steps to update your DB instance and the Seqera installation.
The database volume is persistent on the local machine by default if you use the volumes
key in the db
or redis
section of your docker-compose.yml
file to specify a local path to the DB or Redis instance. If your database is not persistent, you must back up your database before performing any application or database upgrades.
To upgrade your database schema:
groundswell
database resides in a database instance separate from your Seqera database, make a backup of your groundswell
database as well.groundswell
database) in a database separate from your Seqera database, update the MySQL image for your groundswell
database instance while the application is down (during step 4 or 5 above). If you're using the same database instance for both, the groundswell
update will happen automatically during the Seqera database update.Custom deployment:
/migrate-db.sh
script provided in the migrate-db
container. This will migrate the database schema.If you must host your nf-launcher container image on a private image registry, copy the nf-launcher image to your private registry. Then update your tower.env
with the launch container environment variable:
TOWER_LAUNCH_CONTAINER=<FULL_PATH_TO_YOUR_PRIVATE_IMAGE>
If you're using AWS Batch, you will need to configure a custom job definition and populate the TOWER_LAUNCH_CONTAINER
with the job definition name instead.
We're excited to announce that Tower is now Seqera Platform. This name change underscores our vision to evolve Seqera as a single platform for the scientific data analysis lifecycle.
While the underlying platform remains the same, over time you can expect Seqera to become even more scalable, flexible and capable. In the coming weeks and months, references to Tower will be replaced across our product documentation and communications.
We're pleased to announce the availability of Seqera Enterprise 23.3, an important first step in delivering on this revamped product vision and roadmap. Seqera 23.3 includes significant new functionality, including a new Data Explorer, enhanced support for Google Cloud Batch and Google Life Sciences, and much more.
Data Explorer is a powerful new feature of the Seqera platform that lets you easily visualize, search for, and manage data across different cloud providers. This enables you to easily link data to pipelines, troubleshoot runs, and examine outputs - all without switching context. Actions such as file preview, download and upload, as well as custom bucket creation and deletion are logged and details can be accessed in the admin panel.
Data Explorer addresses the scientific community's need to streamline data management for pipelines, from arrival in cloud storage, to diving into the different outputs of a pipeline, and passing data to downstream analysis. We started simplifying this process with datasets, a convenient metadata layer to organize versioned, structured data. Data Explorer is the next big step to enable users to manage their data and analyses in one simple workflow.
Data Explorer simplifies data management across multiple cloud object stores, including Amazon S3, Azure Blob Storage, and Google Cloud Storage. With Data Explorer, organizations can:
Data Explorer is accessible via the new Data Explorer tab in Seqera Platform. You can also access the interface to upload files or select datasets and destination storage buckets for pipeline runs.
Seqera uses secrets to store the keys and tokens used by workflow tasks to interact with external systems, e.g., a password to connect to an external database or an API token. Seqera relies on third-party secret manager services to maintain security between the workflow execution context and the secret container. This means that no secure data is transmitted from Seqera to the compute environment.
In Seqera 23.3, you can now take advantage of secrets in Google Cloud Batch or Google Life Sciences compute environments by using Google Secrets Manager as the underlying user secrets store.
Pipeline resource optimization allows you to minimize the resources used in your pipeline runs based on the resource use of previous runs.
When a run completes successfully, Seqera automatically creates an optimized profile for it. This profile consists of Nextflow configuration settings for each process and each of the following resource directives (where applicable): cpus
, memory
, and time
. The optimized setting for a given process and resource directive is based on the maximum use of that resource across all tasks in that process.
Login redirection logic update
Login redirection logic has changed in version 23.3. Seqera now prepends the TOWER_SERVER_URL
(or tower.serverUrl
in tower.yml
configuration) to the authentication redirect URL during the login flow. This is useful when your server URL contains a contextual path.
If you specify a DNS name as your TOWER_SERVER_URL
, but access your Seqera instance using a different address (such as using an IP address that resolves to the server URL asynchronously), user login will not resolve.
Revert default Tower name changes in documentation
A previous iteration of the rebranded Seqera documentation noted seqera
as the default and example value for certain variables (such as default database names). The rebranding from Nextflow Tower to Seqera Platform is an ongoing, incremental process and as such, legacy tower
values and naming conventions used by the Seqera backend will remain in place until a future release. Updates to configuration variables and values will be communicated well in advance to prepare users for any breaking changes.
This version requires a database schema update. Follow these steps to update your DB instance and the Seqera installation.
To ensure no data loss, the database volume must be persistent on the local machine. Use the volumes
key in the db
or redis
section of your docker-compose.yml
file to specify a local path to the DB or Redis instance.
Docker Compose:
docker compose down
, then docker compose up
.Kubernetes:
kubectl apply -f tower-cron.yml
. This will automatically migrate the database schema.kubectl apply -f tower-srv.yml
.Custom deployment:
/migrate-db.sh
script provided in the backend
container. This will migrate the database schema.If you must host your nf-launcher container image on a private image registry, copy the nf-launcher image to your private registry. Then update your tower.env
with the launch container environment variable:
TOWER_LAUNCH_CONTAINER=<FULL_PATH_TO_YOUR_PRIVATE_IMAGE>
If you're using AWS Batch, you will need to configure a custom job definition and populate the TOWER_LAUNCH_CONTAINER
with the job definition name instead.
Leveraging the Fusion file system, you can now run the Nextflow head job of your pipelines with the AWS Fargate container service. Fargate is a serverless compute engine compatible with Amazon ECS that enables users to run containers without the need to provision servers or clusters in advance. The scalable Fargate container service can help speed up pipeline launch and reduce cloud-related costs by minimizing the time for cloud infrastructure to be deployed.
BEST_FIT_PROGRESSIVE
for on-demand CEs and SPOT_CAPACITY_OPTIMIZED
for spot CEs.before:
search keywords from Dashboard to runs page.syscr
and syscw
in task details modal.Breaking changes and instructions listed here apply when updating from Tower version 23.1. If you are updating from an earlier version, see the release notes of previous versions for a complete picture of changes that may affect you.
Several new Tower features over the last few releases require updated AWS IAM permissions policies. Retrieve and apply the latest policy files here.
The Wave containers service uses container registry credentials in Tower to authenticate to your (public or private) container registries. This is separate from your existing cloud provider credentials stored in Tower.
This means that, for example, AWS ECR (Elastic Container Registry) authentication requires an ECR container registry credential if you are running a compute environment with Wave enabled, even if your existing AWS credential in Tower has IAM access to your ECR.
See the relevant container registry credentials page for provider-specific instructions.
This Tower version requires a database schema update. Follow these steps to update your DB instance and the Tower installation.
!!! warning ""
To ensure no data loss, the database volume must be persistent on the local machine. Use the volumes
key in the db
or redis
section of your docker-compose.yml file to specify a local path to the DB or Redis instance.
Make a backup of the Tower database.
Download and update your container versions.
Redeploy the Tower application:
docker compose:
docker compose down
, then docker compose up
.kubernetes:
kubectl apply -f tower-cron.yml
. This will automatically migrate the database schema.kubectl apply -f tower-srv.yml
.custom deployment:
/migrate-db.sh
script provided in the backend
container. This will migrate the database schema.If you must host your nf-launcher container image on a private image registry:
Copy the nf-launcher image to your private registry.
Update your tower.env
with the launch container environment variable:
TOWER_LAUNCH_CONTAINER=<FULL_PATH_TO_YOUR_PRIVATE_IMAGE>
!!! warning ""
If you're using AWS Batch, you will need to configure a custom job definition and populate the TOWER_LAUNCH_CONTAINER
with the job definition name instead.
Share your feedback via support.seqera.io.
To enhance pipeline search and navigation capabilities, we now support a new list view to complement the existing card view. The list view allows users to efficiently search for and navigate to their pipeline of choice, while also ensuring that the most relevant information is visible and the relationships between pipelines are clear. With this new feature, users can access their pipelines in either card or list view, making them easier to manage.
We've also introduced a new pipeline detail view that shows in-depth information about each pipeline without needing to access the edit screen.
Tower 23.1 introduces support for the Fusion file system in Google Cloud Batch environments. Fusion is a distributed, lightweight file system for cloud-native pipelines that has been shown to improve performance by up to ~2.2x compared to cloud native object storage.
With this new integration, Google Cloud Batch users can enjoy a faster, more efficient, and cheaper processing experience. Fusion offers many benefits, including faster real-time data processing, batch processing, and ETL operations, making it a valuable tool for managing complex data pipelines. By using Fusion with Google Cloud Batch, users can run their data integration workflows directly against data residing in Google Cloud Storage. This integration will allow Google users to streamline their data processing workflows, increase productivity, reduce cloud spending, and achieve better outcomes.
We have added a new secure way to connect two elements, Tower and Wave, using WebSockets. This is an important addition for our enterprise customers as it ensures connection safety, improved efficiency, and better control over traffic sent between Tower and Wave. This connection will help facilitate the adoption of Fusion by enterprise customers, as it provides a more secure and reliable way to manage their data integration workflows. With WebSockets, users can easily connect their Tower and Wave instances and take advantage of the many benefits that Fusion has to offer.
Breaking changes and instructions listed here apply when updating from Tower version 22.4. If you are updating from an earlier version, see the release notes of previous versions for a complete picture of changes that may affect you.
Several new Tower features over the last few releases require updated AWS IAM permissions policies. Retrieve and apply the latest policy files here.
The Wave containers service uses container registry credentials in Tower to authenticate to your (public or private) container registries. This is separate from your existing cloud provider credentials stored in Tower.
This means that, for example, AWS ECR (Elastic Container Registry) authentication requires an ECR container registry credential if you are running a compute environment with Wave enabled, even if your existing AWS credential in Tower has IAM access to your ECR.
See the relevant container registry credentials page for provider-specific instructions.
This Tower version requires a database schema update. Follow these steps to update your DB instance and the Tower installation.
!!! warning ""
To ensure no data loss, the database volume must be persistent on the local machine. Use the volumes
key in the db
or redis
section of your docker-compose.yml
file to specify a local path to the DB or Redis instance.
Make a backup of the Tower database.
Download and update your container versions.
Redeploy the Tower application:
docker compose:
docker compose down
, then docker compose up
.kubernetes:
kubectl apply -f tower-cron.yml
. This will automatically migrate the database schema.kubectl apply -f tower-srv.yml
.custom deployment:
/migrate-db.sh
script provided in the backend
container. This will migrate the database schema.If you must host your nf-launcher container image on a private image registry, copy the nf-launcher image to your private registry. Then update your tower.env
with the launch container environment variable:
TOWER_LAUNCH_CONTAINER=<FULL_PATH_TO_YOUR_PRIVATE_IMAGE>
!!! warning ""
If you're using AWS Batch, you will need to configure a custom job definition and populate the TOWER_LAUNCH_CONTAINER
with the job definition name instead.
Share your feedback via support.seqera.io.