# Seqera Docs > Complete reference documentation for Seqera Platform, Nextflow, MultiQC, Fusion, and Wave. Use this file for deep technical lookups across all Seqera products. It contains the full content of every documentation page, including API references, configuration options, and CLI commands. # Seqera Platform Enterprise > Documentation for Seqera Platform Enterprise. This file contains all documentation content in a single document following the llmstxt.org standard. ## Admin panel As a root user, you can access a comprehensive overview of the organizations, workspaces, users, and teams in your account from the **Admin panel**. It also includes tabs for application event audit logs, administrative statistics, and system configuration options. The root user system role should only be assigned to a system administrator as it provides high-level visibility and configuration access to your account. :::tip See [Basic configuration](../enterprise/configuration/overview.mdx#basic-configuration) to learn how to add root users to your Seqera Enterprise deployment with the `TOWER_ROOT_USERS` environment variable. ::: ## Organizations :::note From version 23.2, organization owners and root users can edit organization names on the **Edit organization** page. ::: The **Organizations** tab lists all the organizations in your account. - Use the search function to find an organization by name and perform various operations with that organization. - Select **Add organization** to create a new organization. - Select an organization name from the table to edit or delete it. See [Organizations](../orgs-and-teams/organizations) for more information. ### Members From an organization's page, select the **Members of organization** tab to view a list of its members. Here you can list and search for all members and owners of the organization, change a member's role, remove a member from the organization, or add a new member to the organization. You can only add existing users to an organization. You can't remove the last owner of an organization until you promote another member to **Owner** first. See [Members](../orgs-and-teams/organizations#members) for more information. ### Quotas Each organization has usage quotas, such as the maximum number of pipeline runs that can be active (submitted or running) at the same time. In Seqera Platform Enterprise, your Seqera license defines these quotas. You can't view or edit them in the **Admin panel**. [Contact Seqera support](https://support.seqera.io) to review or change the quotas for your organizations. If a pipeline launch fails with `Limit of N running workflows reached`, see [Troubleshooting](../troubleshooting_and_faqs/troubleshooting#limit-of-n-running-workflows-reached). ## Workspaces The **Workspaces** tab lists all the workspaces in your account. - Use the search function to find a workspace by name to view and edit that workspace. - Select **Add workspace** to create a new workspace. Choose a workspace name that isn't already in use. If the new workspace name already exists in the system, the creation will fail. After the workspace is created, it's listed in the **Workspaces** tab. - Select **Edit** next to a workspace name to edit or delete the workspace. See [Workspaces](../orgs-and-teams/workspace-management) for more information. ## Users The **Users** tab lists all the users in your account. - Select **Add user** to create a new user. If the new user email already exists in the system, the user creation will fail. After the user is created, inform them that access has been granted. - Use the search function to find a user by name or email. - Select a username from the list or select **Edit** to view and update the user's details. - To disable a user's Platform login access, select **Disable user**. This action does not delete the user. - To reinstate a disabled user's Platform login access, select **Enable user**. - To give login access to a user who does not have it yet, select **Allow login**. This option is grayed out for users who already have login access. See [User roles](../orgs-and-teams/roles) for more information on organization and workspace user access roles. ## Teams The **Teams** tab lists all the teams in your account. - Use the organizations drop-down next to the search bar to filter teams by organization. - Use the search function to find a team by name and perform various operations. - Select **Add team** to create a new team. - Select **Edit** next to a team to edit the team's details, or select **Delete** to delete it. From the teams list, you have an overview of the number of members and the unique ID of each team. Select **Edit** to view a team's page, or select the number next to **Members:** to go to the **Members** tab of the team page. From the **Members of team** tab, you can list and search for all users that are members of the selected team, change a user's role, remove a member from the team, or add a new member to the team. See [Teams](../orgs-and-teams/organizations#teams) for more information. ## Audit logs View application event [audit logs](../monitoring/audit-logs). ## Encryption With [secret key rotation](../enterprise/configuration/overview.mdx#secret-key-rotation) configured in your Enterprise instance, the **Encryption** tab displays the status of encryption tasks as they complete. Encryption tasks complete with one of three statuses: - **Success**: Secret encryption with the new secret key has completed successfully. - **Error**: Secret encryption has failed, usually due to corrupted or missing data. **Encrypted data ID** displays the path to the secret or credential. - **Orphaned**: While secret data has been successfully encrypted, the secret is orphaned in the Platform database. This means that this secret data exists in an encrypted state in your Platform database without a corresponding secret or credential ID in a Platform workspace. **Encrypted data ID** displays the unique identifier for the orphaned entry in your Platform database. --- ## Agents An agent is a reusable, named set of instructions that Co-Scientist runs on your behalf in a Seqera Platform workspace. A Co-Scientist conversation is interactive and starts empty each time. An agent captures a task you repeat, such as investigating failed runs or summarizing results. Anyone in the workspace can then run it without rewriting the prompt. Agents are workspace-scoped. Every agent in a workspace is visible to everyone with permission to read agents, not only the person who created it. An agent acts as the [service account](#agent-identity-and-permissions) bound to it rather than as the person who starts it. It reaches private repositories through a [GitHub App credential](#access-to-private-git-repositories) in its workspace. ## Enable agents Agents are available from Seqera Platform Enterprise 26.2. Once `TOWER_AGENT_BACKEND_URL` is set, they are enabled in every organization unless an administrator restricts them. | Environment variable | Effect | |---|---| | `TOWER_AGENT_BACKEND_URL` | Origin of the Co-Scientist agent backend. While it is empty, agents are off in every organization, whatever the allow-list says. The Platform Helm chart sets it when the `agent-backend` subchart is enabled. Chart 1.0.4 enables that subchart by default. Set it by hand only if you deploy some other way. See [Co-Scientist configuration](../enterprise/configuration/overview#co-scientist). | | `TOWER_AGENT_CONFIGURATION_ALLOWED_ORGANIZATIONS` | Which organizations have agents. Leave it unset, or set it empty, to enable agents in every organization. Set it to a comma-separated list of organization IDs to enable them for those organizations only. | Agents are organization-scoped. They are never available in a personal workspace, whatever the allow-list says. The allow-list controls the **Agents** pages, the **Trigger agent** button, [Actions that target an agent](../pipeline-actions/overview#agent-targets), and the agents API. In an organization where agents are off, the `/agents` endpoints return `404`. The same variables govern [service accounts](../orgs-and-teams/create-service-accounts). Enabling agents for an organization also enables service accounts for it. With service accounts enabled, you can configure an agent's identity. :::note GitHub App credentials need no configuration in Enterprise. They are available in every workspace. ::: ## Agents, chat, and coding agents Seqera uses "agent" for three things: - **Agents** — reusable instruction sets you create in a workspace and run against your pipelines, described here. - **[Co-Scientist chat](./platform.md)** — the interactive assistant panel in Seqera Platform. - **[Coding agents](./coding-agents.md)** — third-party tools such as Claude Code or Codex, which you connect to Seqera through a skill. Agents are also unrelated to Tower Agent, the component that connects Seqera Platform to an HPC cluster. ### Agents compared with Co-Scientist chat Agents and Co-Scientist chat are the same assistant reached two ways. Both work inside a Seqera Platform workspace and can read your runs and act on them. They differ in who they act as, who sees them, and what starts them. | | Agent | Co-Scientist chat | |---|---|---| | Where it runs | In the background, once started | In the Co-Scientist panel in Seqera Platform, while you watch | | Instructions | Written once and saved, reused on every run | Written fresh in each conversation | | Acts as | The service account bound to the agent | You, with your own permissions | | Who sees it | Everyone in the workspace who can read agents | Only you, in your own session | | What starts it | You, from a run | You, by typing in the panel | ## Permissions Two things control agents. Workspace grants decide who can see and manage them, and the bound service account decides what an agent can do when it runs. ### Who can manage agents Four workspace grants control access to agents: | Grant | Allows | |---|---| | `Agent_Read` | View and search the workspace's agents | | `Agent_Write` | Create and edit agents, and pause or resume them | | `Agent_Execute` | Run an agent | | `Agent_Delete` | Remove an agent | **Agents** appears in the navigation only for users with `Agent_Read`. ### Agent identity and permissions An agent acts as a service account, not as the person who starts it. Binding one is required. Select it under **Agent permissions** when you create or edit the agent. Every run of that agent then uses that account's roles. Whoever runs it, the agent can only reach what the service account is allowed to reach. - Only service accounts assigned to the agent's workspace can be bound. An account with no role in the workspace cannot act there. - You cannot bind a service account that holds permissions you do not hold in the workspace yourself. - Service accounts belong to the organization. If none is assigned to your workspace, ask an organization owner to [create one and assign it](../orgs-and-teams/create-service-accounts) before you create the agent. - If the bound service account is later removed from the workspace or deleted from the organization, Seqera Platform pauses the agent. It shows as **Inactive** and drops out of **Trigger agent**. Adding the service account back does not resume the agent. Select **Resume** on the **Agents** page. When the person removing the service account is not a participant in the agent's workspace, the agent stays active, and Seqera Platform refuses its next launch instead. - If the bound service account is disabled, the agent's runs fail rather than fall back to the permissions of whoever started them. :::caution Scope the service account to what the agent's instructions need. Every run of the agent carries those permissions, started by anyone in the workspace with `Agent_Execute`. ::: If you do not see an **Agent permissions** section on the agent form, agents are not enabled for your organization. See [Enable agents](#enable-agents). ## Access to private Git repositories An agent reaches a private repository only through a [GitHub App credential](../git/overview.md#github-app). No other Git credential type works for an agent. A personal access token does not, even though it works elsewhere in Seqera Platform. Bind the credential to the agent under **Agent permissions**, alongside the service account. The credential is optional. An agent without one still runs, but cannot clone, commit, or push. With one bound, the agent can do all three, as that credential. Two things have to be in place before you can bind it: - A GitHub organization admin installs the GitHub App in the GitHub organization and grants it access to the repositories the agent needs. - The credential exists in the workspace the agent runs in. Credentials are workspace-scoped. An agent cannot use one from another workspace or from someone's personal credentials. If a bound credential is later removed from the workspace or marked invalid, the agent form warns you and blocks saving until you select a different credential or clear it with **None**. ## Create an agent 1. In your workspace, select **AI** > **Agents**. 1. Select **Add**. If the workspace has no agents yet, select **Add agent** instead. 1. Choose a starting point: - **Fix failed runs**, **Summarize successful runs**, or **Summarize failed runs** — a template that prefills the name, description, and instructions. - **New agent** — write your own instructions. 1. Complete the details: - **Name** (required). Letters, numbers, dashes, and underscores only. - **Description** (optional). - **Agent instructions** (required). What the agent should do when it runs. 1. Under **Agent permissions**: - **Service account** (required). The identity the agent runs as. Only accounts assigned to this workspace are listed. See [Agent identity and permissions](#agent-identity-and-permissions). - **GitHub App credential** (optional). Needed only if the agent works with private Git repositories. See [Access to private Git repositories](#access-to-private-git-repositories). 1. Save the agent. Templates prefill text only. They do not set a trigger, a schedule, or anything about where the agent runs. Review and edit the instructions before saving. ## Manage agents The **Agents** page lists every agent in the workspace that you can read, with its status. From there you can: - **Search** for an agent by name. - **Edit** an agent's name, description, instructions, service account, and GitHub App credential. The template picker is not shown when editing. - **Pause** an agent to keep its definition but stop it being selectable, and **Resume** it later. A paused agent shows as **Inactive**. - **Remove** an agent. Seqera Platform asks you to confirm. Paused and removed agents do not appear when selecting an agent to run. Removing an agent does not delete the sessions it already produced. ## Run an agent You start an agent from a run, so it works with that run's context. Running one needs `Agent_Read` and `Agent_Execute`, and access to [Co-Scientist chat](./platform.md). Chat access requires the panel to be enabled for your organization and the `chat:execute` permission. Without chat, **Trigger agent** does not appear. 1. Open the run, or find it in a project's **Runs** tab. 1. Select **Trigger agent**. 1. Choose an agent from the list. Search by name if the workspace has many. The agent's session opens in the Co-Scientist panel, where you can follow what it does. Agent sessions are read-only. To continue the conversation, select **Fork conversation**, which copies the session into a chat of your own and opens it. Only active agents appear in the list. If the workspace has none, the list offers **Add agent** to users with `Agent_Write`. ## What an agent does not define An agent holds a name, a description, its instructions, and the service account it acts as. It does not carry a trigger, a schedule, a compute environment, or a sandbox configuration. Those are properties of how the agent is run, not of the agent itself. ## Learn more - [Co-Scientist in Seqera Platform](./platform.md): The Co-Scientist panel in Seqera Platform - [Coding agents](./coding-agents.md): Connect Claude Code, Codex, and other agents to Seqera - [Create and manage service accounts](../orgs-and-teams/create-service-accounts): The identities agents run as - [Git integration](../git/overview.md): Connect Seqera to public and private Git repositories - [Usage and cost](./usage-and-cost.md): Co-Scientist usage and cost --- ## Authentication The Seqera CLI uses your Seqera Platform account to authenticate you with Co-Scientist. This page covers how to log in and out, authenticate in automated environments, connect to your Enterprise backend, and how token refresh works. :::info[**Prerequisites**]{#prerequisites} You will need the following to get started: - [Seqera CLI](./installation.mdx) - A user account on your Seqera Platform Enterprise deployment ::: ## Log in To authenticate with your Seqera Platform account, run: ```bash seqera login ``` This will: 1. Open your default browser to the Seqera login page. 1. Prompt you to sign in with your Seqera Platform credentials. 1. Automatically capture the authentication token. 1. Display a success message in your terminal. ```text [Login] Starting Seqera CLI authentication... [Login] ✓ Authentication successful! [Login] ✓ Organization set: ``` ## View session status To view your current session status, use the `/status` command inside the TUI: ```text /status ``` This shows your authentication status and organization details. ## Log out To sign out from the current session, run: ```bash seqera logout ``` This command revokes your current authentication token and removes locally stored credentials. You will need to re-authenticate on next use. ## Organization management The Seqera CLI manages your organization selection for billing. Use the `seqera org` commands to view and switch organizations: - `seqera org`: View your current organization - `seqera org list`: List all organizations - `seqera org switch`: Switch organization - `seqera org clear`: Clear organization selection ## Token refresh The Seqera CLI automatically refreshes your authentication token when needed. You are not required to log in again unless: - You explicitly log out - Your refresh token expires (typically after extended inactivity) - Your Seqera Platform account permissions change ## Add access tokens for automation For automated environments, you can provide a Seqera Platform access token directly using the `SEQERA_ACCESS_TOKEN` environment variable: ```bash export SEQERA_ACCESS_TOKEN= seqera ai ``` When this environment variable is set, the CLI skips the OAuth login flow and uses the provided token directly. ## Connect to an Enterprise backend Set the following environment variables before starting `seqera ai`: | Variable | Purpose | Example value | | --- | --- | --- | | `SEQERA_AI_BACKEND_URL` | Co-Scientist backend endpoint used by the CLI | `https://ai-api.platform.example.com` | | `SEQERA_AUTH_DOMAIN` | OIDC authority base URL. The CLI fetches OpenID configuration from this URL and opens the discovered authorization endpoint in your browser. | `https://platform.example.com/api` | | `SEQERA_AUTH_CLI_CLIENT_ID` | OAuth client ID for the Seqera CLI | `seqera_ai_cli` | | `TOWER_ACCESS_TOKEN` | Platform personal access token used instead of browser login | `` | Use the OAuth login flow: ```bash export SEQERA_AUTH_DOMAIN=https://platform.example.com/api export SEQERA_AUTH_CLI_CLIENT_ID=seqera_ai_cli export SEQERA_AI_BACKEND_URL=https://ai-api.platform.example.com seqera ai ``` Use a Platform personal access token instead of browser login: ```bash export SEQERA_AUTH_DOMAIN=https://platform.example.com/api export TOWER_ACCESS_TOKEN= export SEQERA_AI_BACKEND_URL=https://ai-api.platform.example.com seqera ai ``` To avoid exporting these variables in every shell, set `authDomain` and `backendUrl` in `~/.config/seqera-ai/config.json` instead. See [Configuration file](./reference/environment-variables.md#configuration-file). Run `seqera info` to confirm which Platform and agent backend the CLI uses. Set `SEQERA_AUTH_CLI_CLIENT_ID` only for OAuth deployments that use a non-default CLI client ID. Current CLI builds still require `SEQERA_AUTH_DOMAIN` for Enterprise token-based authentication so the CLI can target the correct Platform authority. ## Learn more - [Co-Scientist](index.md): Co-Scientist overview - [Installation](./installation.mdx): Install, update, and configure the CLI - [Command approval](./command-approval.md): Control which commands run automatically - [Use cases](./use-cases.md): Co-Scientist use cases - [Usage and cost](./usage-and-cost.md): Co-Scientist usage in Enterprise deployments - [Skills](./reference/skills-reference.md): Built-in skills, slash commands, and session limits - [Troubleshooting](../troubleshooting_and_faqs/coscientist_troubleshooting.md): Troubleshoot common errors --- ## Bedrock setup Co-Scientist runs Claude inference in your own AWS account through Amazon Bedrock. Before you install the Helm charts, configure your AWS account so the Co-Scientist pods can invoke the models: enable model access, grant the IAM permissions the agent backend needs, and — if you use sandboxed sessions — create an AgentCore runtime. Complete this page after the [prerequisites](./prerequisites.md) and before [installation](./installation.mdx). :::note In agent backend versions after `1.14.1`, including the version shipped with Enterprise 26.2, Co-Scientist uses a single model for text inference. Earlier agent backends route text inference across separate primary, fast, and deep models and need model access and IAM permissions for each. See the 26.1 documentation if your deployment pins one of those. ::: ## Prerequisites Before you begin, make sure you have: - An AWS account with Amazon Bedrock available in the region you plan to use. - Permission to modify IAM policies and to enable Bedrock model access in that account. - The AWS CLI v2.34.1 or later installed locally. - If you enable sandboxed AgentCore sessions: an Amazon ECR registry in the same account, a local container tool (Docker or Podman), and credentials for `cr.seqera.io`. ## Enable model access Enable access to the models Co-Scientist uses in your chosen region. See the AWS documentation for [adding or removing access to Amazon Bedrock foundation models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access-modify.html). | Purpose | Model ID | Required | | --- | --- | --- | | Text inference | `anthropic.claude-opus-4-8` | Always | | Text embeddings | `amazon.titan-embed-text-v2:0` | Only when documentation semantic search is enabled | Co-Scientist uses a single model for all text inference. Earlier releases routed requests across separate primary, fast, and deep models. That tiering was removed, and only the one model above needs access. Configure the embedding model separately from chat inference, in `bedrock.embeddings.model`. The chart default is `amazon.titan-embed-text-v2:0`. Leave `embeddings.provider` unset to disable documentation search and skip this model entirely. :::note The first time an account requests access to Anthropic models, AWS may ask you to submit use case details before approving access. Approval is not instant — request access before you schedule your installation. ::: ### Activate the Marketplace subscription Models served through AWS Marketplace can only be invoked while an active agreement exists for them. AWS creates the agreement automatically on first use, so activate each model once before you install: - Open the model in the [Bedrock model catalog](https://docs.aws.amazon.com/bedrock/latest/userguide/model-catalog.html) and run it in the playground, or - Invoke it once with the [InvokeModel](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html) or [Converse](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html) API. If you skip this step, Co-Scientist's first inference call fails even though model access is enabled. ## Use an inference profile for the Claude model `Claude Opus 4.8` is not available for on-demand throughput, and you must invoke it through an inference profile. Use the global profile: ```text global.anthropic.claude-opus-4-8 ``` Supply its full ARN to the agent backend as `bedrock.inference.anthropicModel` when you install the chart: ```yaml arn:aws:bedrock:::inference-profile/global.anthropic.claude-opus-4-8 ``` ## Grant Bedrock inference permissions Attach the following policy to the IAM role or user that the agent backend pods use: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream" ], "Resource": [ "arn:aws:bedrock:::foundation-model/anthropic.claude-opus-4-8", "arn:aws:bedrock:::inference-profile/global.anthropic.claude-opus-4-8", "arn:aws:bedrock:::foundation-model/amazon.titan-embed-text-v2:0" ] } ] } ``` Replace `` and `` with your own values. Foundation model ARNs are not region-scoped, so they have no region or account component. Both the inference profile and the foundation model it routes to must be listed, because a request through a profile is authorized against both. Drop the `amazon.titan-embed-text-v2:0` entry if you leave documentation semantic search disabled. If you override `bedrock.embeddings.model` with a different embedding model, list that model's ARN instead. :::caution A model missing from this policy fails with an access-denied error at runtime, not at install time. Verify the policy before you hand the deployment over. ::: ## Set up the AgentCore runtime Complete this section only if you enable sandboxed AgentCore sessions by setting `sandbox.provider: bedrock`. Leave `sandbox.provider` unset to run without sandboxed sessions and skip this section entirely — Bedrock AgentCore is the only sandbox provider currently supported. ### Copy the runtime image into your ECR Seqera publishes the AgentCore runtime image. Copy it into a repository in your own Amazon ECR registry, in the same account as the runtime. :::caution The image must be stored in your own ECR registry. AgentCore cannot pull it from Seqera's container registry — it expects to resolve images through AWS IAM within your account. ::: The image is built for `arm64`. Pull it, retag it for your registry, and push: ```bash podman pull --arch arm64 cr.seqera.io/ai/agent-backend/bedrock-agentcore-runtime: aws ecr get-login-password --region \ | docker login --username AWS --password-stdin .dkr.ecr..amazonaws.com docker tag cr.seqera.io/ai/agent-backend/bedrock-agentcore-runtime: \ .dkr.ecr..amazonaws.com/agent-backend/bedrock-agentcore-runtime: docker push .dkr.ecr..amazonaws.com/agent-backend/bedrock-agentcore-runtime: ``` Pin a specific dated tag rather than `latest`. Contact Seqera for the tag matching your deployment. ### Create the runtime 1. In the AWS console, create a new AgentCore runtime with the **Host agent or tool** option. 2. Set its source to the image in your ECR registry, and accept the defaults for the remaining options. 3. Copy the runtime ARN. It looks like this: ```text arn:aws:bedrock-agentcore:::runtime/- ``` 4. Supply that ARN to the agent backend as `bedrock.sandbox.runtimeArn` when you install the chart. :::note Creating the runtime requires the identity you use for setup to allow `bedrock-agentcore:CreateAgentRuntime` and `bedrock-agentcore:ListAgentRuntimes`. This is separate from the runtime-invocation policy below, which applies to the agent backend pods. Service control policies or permissions boundaries in your organization can block these actions even when your own role allows them. ::: ### Grant runtime invocation permissions Attach the following policy to the IAM role or user that the agent backend pods use: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "bedrock-agentcore:InvokeAgentRuntime", "bedrock-agentcore:InvokeAgentRuntimeCommand", "bedrock-agentcore:InvokeAgentRuntimeForUser", "bedrock-agentcore:InvokeAgentRuntimeWithWebSocketStream", "bedrock-agentcore:InvokeAgentRuntimeWithWebSocketStreamForUser" ], "Resource": [ "arn:aws:bedrock-agentcore:*::runtime/*", "arn:aws:bedrock-agentcore:*::runtime/*/runtime-endpoint/*" ] } ] } ``` ## Provide credentials to the pods Bedrock authentication uses AWS IAM credentials. No API key secret is needed for the Bedrock path. | Method | When to use | | --- | --- | | EKS Pod Identity | Recommended. Associates the IAM role with the agent backend's Kubernetes service account, with no long-lived credentials in the cluster. | | IAM roles for service accounts (IRSA) | Supported alternative on clusters already standardized on IRSA. | | Static AWS credentials | Supported, but stores long-lived keys in a Kubernetes Secret. Use only when neither role-based option is available. | When the pods must assume a role to reach Bedrock, set `bedrock.default.assumeRoleArn` to that role's ARN. Leave it empty when the pods already hold credentials for the target account. Per-service overrides — `bedrock.inference.assumeRoleArn`, `bedrock.embeddings.assumeRoleArn`, and `bedrock.sandbox.assumeRoleArn` — are available when each capability needs a different role. For the full values file and the surrounding chart configuration, see [Install Co-Scientist](../enterprise/install-seqera-coscientist.mdx). ## Next steps - [Install Co-Scientist](../enterprise/install-seqera-coscientist.mdx) - [Usage and cost](./usage-and-cost.md) — how model access, inference profiles, quotas, and IAM roles map to what your organization is billed for --- ## Coding agents The `seqera skill` command installs a skill file that lets your coding agent use Co-Scientist as a subagent. Once installed, the agent can invoke Co-Scientist directly to manage workflows, build containers, query nf-core modules, and more without leaving your environment. ## Supported agents Co-Scientist installs as a skill into any of the following agents. Each guide covers installation, update, and the available `seqera skill` options for that agent: - [Claude Code](./skill-claude-code.md): Install the skill for [Claude Code](https://claude.ai/code) - [Codex](./skill-codex.md): Install the skill for [Codex](https://openai.com/codex) - [GitHub Copilot](./skill-github-copilot.md): Install the skill for [GitHub Copilot](https://github.com/features/copilot) - [Other coding agents](./skill-other-agents.md): Install the skill for Cursor, OpenCode, Pi, Windsurf, and others --- ## Command approval Co-Scientist can execute local commands and edit files in your environment. This page explains approval modes that control which operations run automatically versus which require your permission, including dangerous commands, workspace boundaries, and best practices. :::info Starting a persistent task with `/goal ` switches the session to `full` approval mode automatically so Co-Scientist can continue working without repeated prompts. ::: ## Approval prompts When a command requires approval, you will see a prompt similar to: ```text APPROVAL REQUIRED (default mode) Command: rm -rf ./build/ [1] Yes, approve this command [2] Always approve this session [3] No, reject Press 1, 2, or 3 to choose ``` You can: - **1**: Run the command once (or press Enter) - **2**: Run the command and auto-approve commands that match it for the rest of the session. Co-Scientist stores a pattern derived from the command, such as `git commit*`, not a blanket approval. Other commands still prompt. A session keeps up to 100 approved patterns. - **3**: Reject the command (or press Escape) An approval prompt stays open for 15 minutes. If you don't respond in that time, the request times out and Co-Scientist stops waiting for the command result. ## Approval modes Approval modes control which local commands Co-Scientist can execute automatically and which require your explicit approval. This provides a balance between convenience and safety when working with local files and commands. There are three approval modes: | Mode | Description | Best for | | --- | --- | --- | | **basic** | Only safe, read-only commands run automatically | Maximum security | | **default** | Safe commands and workspace file edits run automatically | Typical development | | **full** | Everything except dangerous commands runs automatically | Experienced users | You can set the approval mode when starting the CLI: ```bash seqera ai --approval-mode full ``` Or change it during a session using the `/approval` TUI command: ```text /approval basic ``` ### Basic **Rule**: Only safe, read-only commands run automatically. Everything else requires approval. This is the most restrictive mode. The assistant can only auto-execute commands that view information without making changes. **Auto-executes**: - `cat` - View file contents - `ls` - List directory contents - `pwd` - Show current directory - `head` - View file beginning - `tail` - View file end - `tree` - Display directory tree - `echo` - Print text (without file redirection) - `date` - Show current date/time - `whoami` - Show current user - `env` - Display environment variables - `printenv` - Print environment variables - `stat` - Show file status - `uptime` - Show system uptime **Requires approval**: All other commands, including file edits, directory creation, and any other command execution. Safe commands that include file redirections (e.g., `echo "hello" > file.txt`) also require approval. **Use when**: You want maximum control and visibility over every action the assistant takes. **Examples**: ```text > Create a new file called test.txt with "hello world" APPROVAL REQUIRED (basic mode) Command: Write ./test.txt [1] Yes, approve this command [2] Always approve this session [3] No, reject ``` ### Default **Rule**: Safe commands and file operations within your workspace run automatically. All other commands require approval. This is the recommended mode for most users. It allows productive workflow while protecting system files and preventing destructive operations. **Auto-executes**: - All safe commands from basic mode (without file redirections) - File edits **within your current workspace**: - Creating files (`touch`, file creation) - Editing files (text modifications) - Creating directories (`mkdir`) - Copying files (`cp` within workspace) - Moving files (`mv` within workspace) **Requires approval**: - File operations **outside your workspace** - All dangerous commands (see below) - Commands with file redirects to paths outside workspace - All other commands (e.g., `curl`, `wget`, `git`, `npm`, `python`, etc.) **Use when**: You're doing typical development work and want convenience without compromising safety. **Examples**: ```text > Create a new file called test.txt with "hello world" Created ./test.txt ``` File creation in the workspace runs automatically. ```text > Edit /etc/hosts APPROVAL REQUIRED (default mode) Command: Edit /etc/hosts [1] Yes, approve this command [2] Always approve this session [3] No, reject ``` Editing outside the workspace requires approval. ### Full **Rule**: Everything runs automatically except explicitly dangerous commands. This is the most permissive mode. Use it when you fully trust the assistant's actions and want minimal interruption. **Auto-executes**: All commands except those on the dangerous list. **Requires approval**: Only dangerous commands (see below). **Use when**: You're an experienced user comfortable with automated command execution, or when working in an isolated/disposable environment. ## Dangerous commands These commands **always require approval** in any mode: | Command | Risk | | --- | --- | | `rm` | Delete files/directories | | `chmod` | Change file permissions | | `chown` | Change file ownership | | `kill` | Terminate processes | | `killall` | Terminate multiple processes | | `pkill` | Kill processes by name | | `sudo` | Execute as superuser | | `dd` | Low-level data operations | | `mount` | Mount filesystems | | `umount` | Unmount filesystems | | `mkfs` | Create filesystems | | `reboot` | Restart system | | `shutdown` | Power off system | **Examples**: ```text > Create files and directories as needed Created ./src/utils.py Created ./tests/test_utils.py Created ./config/settings.json ``` Most operations run without prompts. ```text > Delete the build directory APPROVAL REQUIRED (full mode) Command: rm -rf ./build/ [1] Yes, approve this command [2] Always approve this session [3] No, reject ``` Dangerous commands still require approval. ## Workspace boundaries In **default** mode, the "workspace" is your current working directory and its subdirectories. File operations are evaluated as: - **Inside workspace**: `/path/to/workspace/src/file.txt` - auto-executes - **Outside workspace**: `/etc/config` or `~/other-project/file.txt` - requires approval The workspace is set to your current directory when you start the CLI: ```bash # Workspace is /home/user/my-project cd /home/user/my-project seqera ai ``` ## Best practices - **Start with default mode**: It provides a good balance for most workflows - **Use basic mode for unfamiliar projects**: When exploring new codebases - **Reserve full mode for trusted contexts**: Disposable environments or well-understood tasks - **Review dangerous command prompts carefully**: These commands can have significant impact ## Learn more - [Co-Scientist](index.md): Co-Scientist overview - [Installation](./installation): Detailed installation instructions - [Authentication](./authentication): Log in, log out, and session management - [Use cases](./use-cases.md): Co-Scientist use cases - [Usage and cost](./usage-and-cost.md): Co-Scientist usage in Enterprise deployments - [Skills](./reference/skills-reference.md): Built-in skills, slash commands, and session limits - [Troubleshooting](../troubleshooting_and_faqs/coscientist_troubleshooting.md): Troubleshoot common errors --- ## Using Co-Scientist Use Co-Scientist day to day: explore common tasks, switch between modes, manage sessions, add skills, control command approval, and organize your workspace. ## In this section - [Use cases](./use-cases.md): Common tasks you can do, with example prompts - [Modes](./modes.md): Work in build, plan, and goal modes - [Sessions](./sessions.md): Start, continue, resume, and exit sessions, and run non-interactively - [Skills configuration](./skills.md): Discover, create, and install skills - [Command approval](./command-approval.md): Control which commands run automatically - [Code intelligence](./nextflow-lsp.md): Language-server support for Nextflow, Python, and R - [Projects](./projects.md): Organize workspace resources into projects using Platform labels - [Usage and cost](./usage-and-cost.md): Co-Scientist usage in Enterprise deployments --- ## Co-Scientist Co-Scientist is Seqera's AI assistant for bioinformatics. It builds, runs, and debugs Nextflow pipelines, manages your data, and works with your Seqera Platform resources. You can use Co-Scientist in two places: - **In Seqera Platform**: Open the Co-Scientist panel from any workspace page. Because Co-Scientist reads the page you are on, you can ask about the run, pipeline, or dataset in front of you without describing it first. See [Co-Scientist in Seqera Platform](./platform.md). - **In the Seqera CLI**: Run `seqera ai` in your terminal to work in your local checkout with access to your Platform workspace. See [Installation](./installation.mdx). Both use the same assistant and the same Seqera Platform Enterprise account. Skills, modes, command approval, and the Nextflow language server protocol (LSP) are available only in the CLI. :::note Co-Scientist is not part of a default Seqera Platform Enterprise installation. An administrator must deploy the agent backend and the Seqera Model Context Protocol (MCP) server, and configure an inference provider. See [Prerequisites](./prerequisites.md) and [Install Co-Scientist](../enterprise/install-seqera-coscientist.mdx). ::: ## Get started ### In Seqera Platform 1. Sign in to Seqera Platform and open an organization workspace. 1. Select **Co-Scientist** in the navigation to open the panel. 1. Ask a question about the page you are on, such as why a run failed. See [Co-Scientist in Seqera Platform](./platform.md) for page context, conversation history, and workspace scope. ### In the Seqera CLI 1. Install the Seqera CLI: ```bash npm install -g seqera ``` 1. Point the CLI at your Enterprise deployment: ```bash export SEQERA_AUTH_DOMAIN=https://platform.example.com/api export SEQERA_AI_BACKEND_URL=https://ai-api.platform.example.com ``` 1. Log in to Seqera: ```bash seqera login ``` 1. Start your first session: ```bash seqera ai ``` See [Installation](./installation.mdx) for prerequisites, updates, and the configuration file alternative to environment variables. Then see [Quickstart](./quickstart.md) to walk through your first session. ## What you can do Co-Scientist works across the pipeline lifecycle, from writing Nextflow code to running it on Seqera Platform. ### Develop pipelines Generate Nextflow configurations and pipeline schemas, convert scripts from other languages (WDL, R) to Nextflow, and discover over 1,000 nf-core modules with ready-to-run commands. Build reproducible Wave containers from conda or pip packages without writing a Dockerfile. The Nextflow language server detects errors as you edit and lets Co-Scientist navigate Nextflow, Python, and R files. ### Run and debug on Platform Launch, monitor, and debug pipeline runs with real-time status, logs, and run metrics. Browse cloud storage through data-links, manage datasets, generate upload and download URLs, and access reference genomes. Co-Scientist works with the compute environments, datasets, and workspaces your account can already access. ### Sessions, skills, and modes Ask in plain English in Seqera Platform or the CLI. In the CLI, use reusable [skills](./skills.md) exposed as slash commands in the `/` palette. Switch between [build, plan, and goal modes](./modes.md) to match execution, analysis, or long-running tasks. Resume earlier sessions with `seqera ai -c`, and organize workspace resources into [projects](./projects.md) using Platform labels. ## Learn more - [Co-Scientist in Seqera Platform](./platform.md): Use the Co-Scientist panel in Seqera Platform - [Prerequisites](./prerequisites.md): What your installation needs before deploying Co-Scientist - [Install Co-Scientist](../enterprise/install-seqera-coscientist.mdx): Deploy the agent backend and MCP server - [Installation](./installation.mdx): Install, update, and configure the CLI - [Quickstart](./quickstart.md): Run your first Co-Scientist session in the CLI - [Authentication](./authentication.md): Log in, log out, and manage sessions - [Use cases](./use-cases.md): Seqera CLI use cases - [Using Co-Scientist](./configuration.md): Configure modes, sessions, skills, command approval, and more - [Coding Agents](./coding-agents.md): Install Co-Scientist as a skill in your coding agent - [Projects](./projects.md): Group workspace resources with Platform labels - [Usage and cost](./usage-and-cost.md): Co-Scientist usage in Enterprise deployments - [Skills](./reference/skills-reference.md): Built-in skills, slash commands, and session limits - [Troubleshooting](../troubleshooting_and_faqs/coscientist_troubleshooting.md): Troubleshoot common errors --- ## Installation The Seqera CLI runs in your terminal on macOS, Linux, or Windows (via WSL). It connects to Seqera Platform and to the Co-Scientist backend in your Enterprise deployment so you can build, run, and debug Nextflow pipelines from a single interactive session. This page covers how to install, update, and uninstall the CLI with npm, how to switch to a development build, and how to point the CLI at the Co-Scientist backend. Once the CLI is on your PATH, see the [Quickstart](./quickstart.md) to log in and start your first session. :::info[**Prerequisites**]{#prerequisites} You will need the following to get started: - Node.js 18 or later - macOS, Linux, or Windows with WSL - A user account on your Seqera Platform Enterprise deployment - The agent backend URL for your deployment, from your Seqera Platform administrator - Network access to the Co-Scientist agent backend ::: ## Install the CLI Install the CLI globally from the [`seqera` npm package](https://www.npmjs.com/package/seqera): ```bash npm install -g seqera ``` Then confirm the CLI is on your PATH: ```bash seqera --version ``` ### Install a development build To install the latest pre-release, run: ```bash npm install -g seqera@dev ``` The `@dev` tag tracks the latest pre-release CLI. Use it only to test unreleased features. Otherwise install the default tag. ## Update the CLI To check whether a newer version is available, run: ```bash seqera update ``` To update the CLI to the latest published version, run: ```bash npm update -g seqera ``` If you use Co-Scientist as a skill for a coding agent, sync your installed skills with the new CLI version after upgrading: ```bash seqera skill check --update ``` This scans both local and global installations by default. Pass `--global` or `--local` to narrow the scope. ## Uninstall the CLI To remove the CLI from your system, run: ```bash npm uninstall -g seqera ``` To also remove stored credentials and configuration, delete `~/.config/seqera-ai/`. ## Configure the Co-Scientist backend The CLI connects to Seqera Platform Cloud by default. Point it at your Enterprise deployment with environment variables or a configuration file. ### Environment variables Set the agent backend URL and the OIDC authority base URL before starting Co-Scientist: ```bash export SEQERA_AI_BACKEND_URL=https://ai-api.platform.example.com export SEQERA_AUTH_DOMAIN=https://platform.example.com/api ``` The CLI fetches OpenID configuration from `SEQERA_AUTH_DOMAIN` and opens the discovered authorization endpoint in your browser. ### Configuration file To avoid exporting variables in every shell, create `~/.config/seqera-ai/config.json`: ```json { "authDomain": "https://platform.example.com/api", "backendUrl": "https://ai-api.platform.example.com" } ``` Environment variables take precedence over the configuration file. ### Verify the connection Confirm that the CLI resolves your Enterprise deployment: ```bash seqera info ``` `seqera info` shows the authenticated user, the agent backend, and the Seqera Platform endpoint the CLI uses. See [Authentication](./authentication.md#connect-to-an-enterprise-backend) for OAuth and token-based examples, and [Environment variables](./reference/environment-variables.md) for the full reference. ## Learn more - [Quickstart](./quickstart.md): Run your first Co-Scientist session - [Authentication](./authentication.md): Log in, log out, and manage sessions - [Coding agents](./coding-agents.md): Install Co-Scientist as a skill in your coding agent - [Using Co-Scientist](./configuration.md): Configure modes, sessions, skills, command approval, and more - [Usage and cost](./usage-and-cost.md): Co-Scientist usage in Enterprise deployments - [Troubleshooting](../troubleshooting_and_faqs/coscientist_troubleshooting.md): Troubleshoot common errors --- ## Modes Co-Scientist offers three modes that control how much autonomy it has in a session. Choose the right level for each task with [Build mode](#build-mode), [Plan mode](#plan-mode), and [Goal mode](#goal-mode). ## Build mode Build mode is the default interactive mode. Co-Scientist can: - Read and search files - Execute commands - Edit or create files - Carry out workflow changes directly in your workspace Use build mode for implementation work, debugging, code generation, and file edits. ## Plan mode Plan mode is optimized for analysis and implementation planning. In plan mode, Co-Scientist focuses on: - Understanding the problem - Comparing approaches and trade-offs - Producing a step-by-step implementation plan - Reading files and searching code for context Plan mode blocks write and execution tools, including: - `execute_bash_local` - `run_nextflow_lint_local` - `run_nf_core_lint_local` - `write_file_local` - `edit_file_local` - `create_directory_local` If the assistant tries to use one of these tools, the request is rejected and the assistant is told to switch back to build mode. For example: ```text Compare whether I should add FastQC or fastp as the first QC step in this RNA-seq pipeline, including the workflow changes each option would require ``` ```text Plan the work to add GPU support to this pipeline ``` ```text Inspect this repository and outline the changes needed for Seqera Platform deployment ``` ## Switch between build mode and plan mode Toggle modes during a session with `Shift+Tab`. You can also: - Check the current mode in the composer footer. - Run `/status` to view the current mode alongside session and LSP status. - Use `/help` to see mode-aware command guidance. ## Goal mode Goal mode is a persistent workflow for longer tasks. Set a goal with: ```bash /goal ``` For example: ```text /goal migrate this pipeline to DSL2 and add nf-tests ``` ```text /goal update this workflow for AWS Batch and verify the config ``` When goal mode is active, Co-Scientist: - Keeps working toward the same objective over multiple model attempts. - Automatically continues if more work is needed. - Stops when the goal is complete or the goal attempt limit is reached. - Switches approval mode to `full` so work can continue without repeated prompts. Goal mode commands: - `/goal` - `/goal off` Run `/goal` without arguments to inspect the current goal. Run `/goal off` to disable goal mode. Co-Scientist currently gives goal mode up to **3 model attempts** before it stops and asks you to start a new goal. ## Keyboard shortcuts | Shortcut | Action | | --- | --- | | `Shift+Tab` | Toggle between build mode and plan mode. | | Ctrl+Enter | If your terminal supports it, interrupt the current response and send a queued follow-up immediately. | | `Esc` | Clear a queued follow-up or interrupt the current response. | | `Shift+Enter` | Insert a newline in your message. | | `Ctrl+J` | Insert a newline in your message. Use it in terminals where `Shift+Enter` does not work. | ## Learn more - [Sessions](./sessions.md): Start, continue, resume, and exit sessions - [Skills configuration](./skills.md): Discover, create, and install skills - [Command approval](./command-approval.md): Control which commands run automatically - [Use cases](./use-cases.md): Seqera CLI use cases - [Usage and cost](./usage-and-cost.md): Co-Scientist usage in Enterprise deployments - [Skills](./reference/skills-reference.md): Built-in skills, slash commands, and session limits --- ## Code intelligence When you ask Co-Scientist to help with code in your workspace, it uses language server (LSP) context to provide: - Explanations for errors and warnings in your code. - Context-aware completions and suggestions. - Better navigation and understanding across project files. For Nextflow projects, this includes diagnostics and code intelligence for scripts and config files. ## Language support | LSP Server | Extensions | Requirements | | --- | --- | --- | | Nextflow | `.nf`, `.config` | Java 17+ installed | | Python (Pyright) | `.py`, `.pyi` | Auto-installs | | R | `.r`, `.R`, `.rmd`, `.Rmd` | R runtime installed | LSP servers automatically start when you work with files that match these extensions. ## Workspace detection Co-Scientist detects the relevant language context from your active workspace and applies matching intelligence automatically. This means you can move between Nextflow, Python, and R files in the same project and get language-aware assistance without manual setup. See [Nextflow Language Server](https://github.com/nextflow-io/language-server) for advanced configuration details. ## Learn more - [Co-Scientist](./index.md): Co-Scientist overview - [Quickstart](./quickstart.md): Start using Co-Scientist - [Use cases](./use-cases.md): Seqera CLI use cases - [Usage and cost](./usage-and-cost.md): Co-Scientist usage in Enterprise deployments - [Skills](./reference/skills-reference.md): Built-in skills, slash commands, and session limits --- ## Co-Scientist in Seqera Platform The Co-Scientist panel is an assistant docked inside Seqera Platform. It opens beside the page you are working on and reads that page as context. Ask about a run, pipeline, dataset, or data-link without describing it first. :::info[**Prerequisites**]{#prerequisites} You will need the following to get started: - Co-Scientist deployed for your Seqera Platform Enterprise installation. See [Install Co-Scientist](../enterprise/install-seqera-coscientist.mdx). - An organization workspace. The panel is not available in personal workspaces. - A workspace role that includes the `chat:execute` permission. The predefined Owner, Admin, Maintain, Launch, Connect, and Project roles include it. The View role does not. Grant it to custom roles explicitly. See [Custom roles](../orgs-and-teams/custom-roles.md#ai). ::: Once Co-Scientist is deployed, the panel is enabled for every organization in the installation unless your administrator restricts it with `TOWER_AI_CHAT_ALLOWED_ORGANIZATIONS`. If you do not see **Co-Scientist** in the navigation bar, contact your Seqera Platform administrator. ## Open the panel 1. Sign in to Seqera Platform and open an organization workspace. 1. Select **Co-Scientist** in the top navigation bar. Your work stays on screen beside the panel. Close the panel to return to where you were, or keep navigating Seqera Platform while the conversation stays open. ## Page context Co-Scientist reads the page you are viewing and sends that context with each message. The panel shows the current page in a **Viewing:** line above the message box. The context includes: - The organization and workspace you are working in. - The primary resource in view, such as a run, pipeline, dataset, data-link, compute environment, or Studio. - A screenshot of the page content area. Co-Scientist scopes its Platform requests to the current organization and workspace, and acts within that workspace unless you ask it to look elsewhere. Because the panel already has this context, you can ask questions that refer to the page directly: - "Why did this run fail?" - "What does this pipeline do?" - "Summarize the results in this dataset." ### Screenshots The screenshot captures the main content area of the page. It excludes the navigation, the side menu, and the Co-Scientist panel itself. It includes everything else on the page, such as run logs, parameter values, and resource names. The current page's screenshot appears as a chip above the message box. Select the chip to preview it. Remove the chip before sending if you do not want the screenshot attached to that message. :::note Co-Scientist sends page context and screenshots to the agent backend and to the inference provider your administrator configured for Co-Scientist. ::: ## Reference resources with `@` Type `@` in the message box to reference a Platform resource by name and attach it to your message. You can reference pipelines, runs, data-links, datasets, compute environments, and Studios. Recently used resources appear first. Use this to bring in a resource other than the one on screen. ## Conversations and history Each conversation is a separate thread. Open the conversation history to search past conversations by title and return to earlier work. The history separates your own chats from agent sessions. Deleting a conversation from the history removes it from your account. Seqera Platform keeps Co-Scientist conversations for 180 days by default. See [Sessions](./sessions.md#session-retention-and-limits). ## Background work A request keeps running after you close the panel, navigate away, or switch workspaces. Reopen the conversation to see the result. A response still in progress resumes streaming when you reopen it. The workspace owns conversations that background agents start, and they are read-only in the panel. To continue one, fork it into your own conversation. ## Work with GitHub repositories Co-Scientist can clone GitHub repositories, create branches, push commits, and open pull requests. It reaches private repositories through a GitHub App that your administrator configures for your installation. You connect your own GitHub account to the app the first time Co-Scientist needs access to a private repository. GitHub access is available only when your administrator configures it, together with a sandbox provider. See [GitHub access](../enterprise/install-seqera-coscientist.mdx#github-access). This connection is separate from the [GitHub App credentials](../git/overview.md#github-app) that Seqera Platform uses to launch pipelines. To connect GitHub: 1. Ask Co-Scientist to work with a private repository. If you have not connected GitHub, Co-Scientist replies with a link to authorize the GitHub App. 1. Open the link and approve the authorization on GitHub. The page that opens shows a connection ID. 1. Copy the connection ID and paste it into the conversation to finish connecting. 1. If the repository belongs to a GitHub organization where the app is not installed, Co-Scientist returns an install link for the app. Ask an organization admin to install the app and grant it access to the repository, then ask Co-Scientist to retry. The authorization link and the connection ID expire after a short time. If either expires, ask Co-Scientist to start again. :::note If your GitHub organization enforces SAML single sign-on (SSO), GitHub rejects access until you authorize the connection for that organization. Co-Scientist tells you when this happens. Open the authorization link from GitHub, approve access for the organization, and retry. ::: ## Download session files Some requests produce files in the Co-Scientist sandbox, such as a generated configuration or a converted pipeline. When a session has produced files, select **Download sandbox contents** above the message box to download them as a compressed archive. The sandbox is available only when your administrator configures a sandbox provider for Co-Scientist. ## Learn more - [Co-Scientist](./index.md): Overview and where Co-Scientist is available - [Projects](./projects.md): Group workspace resources with Platform labels - [Usage and cost](./usage-and-cost.md): Co-Scientist usage in Enterprise deployments - [Install Co-Scientist](../enterprise/install-seqera-coscientist.mdx): Deploy the agent backend and MCP server - [Troubleshooting](../troubleshooting_and_faqs/coscientist_troubleshooting.md): Troubleshoot common errors --- ## Prerequisites ## Overview Everything you need to have in place before installing Co-Scientist. Complete these requirements, then follow [Bedrock setup](./bedrock-setup.md) to configure your AWS account. :::caution Co-Scientist requires Seqera Platform Enterprise 25.3.6 or later. The Co-Scientist panel in Seqera Platform requires Enterprise 26.2 or later. Co-Scientist is available only on AWS. ::: Co-Scientist is a conversational AI interface for Seqera Platform, available in the [Co-Scientist panel](./platform.md) in Seqera Platform and in the Seqera CLI. Seqera Platform serves the panel itself. You do not deploy a separate web interface. Deploy the following components in sequence: | Order | Component | Purpose | | --- | --- | --- | | 1 | MCP server | Model Context Protocol server providing Platform-aware tools (workflows, datasets, compute environments). Deploy first — the agent backend connects to it at startup. | | 2 | MySQL database | Dedicated database for session state and conversation history. | | 3 | Redis | Caching and session management layer for the agent backend. | | 4 | Agent backend | FastAPI service that orchestrates AI interactions between the CLI, the Co-Scientist panel in Seqera Platform, Bedrock, and MCP. | ## Platform - Co-Scientist is in Early Access for Platform Enterprise and may require an Enterprise version upgrade. [Contact Seqera support](https://support.seqera.io) for more information. - **OIDC** configured in Platform for authentication. ## AWS account Co-Scientist can use Claude models via [Amazon Bedrock](https://aws.amazon.com/bedrock/) or via an Anthropic API key. You need an AWS account with Bedrock available in your chosen region. ### Models The following Bedrock model access must be enabled in your account: | Purpose | Model ID | Required | | --- | --- | --- | | Text inference | `anthropic.claude-opus-4-8` | Always | | Text embeddings | `amazon.titan-embed-text-v2:0` | Only when documentation semantic search is enabled | Co-Scientist uses a single model for all text inference. Agent backend versions up to and including `1.14.1` route requests across separate primary, fast, and deep models and need access to each. See the 26.1 documentation if your deployment pins an earlier version of the agent-backend images. If you use a recent Anthropic model through AWS Bedrock, such as `anthropic.claude-opus-4-8`, make sure your account has access to it through the Bedrock service in your chosen region. Some AWS accounts have additional account-level eligibility requirements for certain models and can return errors like `anthropic.claude-opus-4-8 is not available for this account`. These requirements aren't visible in the Service Quotas console. To test them, use the AWS Bedrock Playground in the console. If your account has these requirements, contact AWS Support to get access to the required models, as explained in [this AWS blog post](https://repost.aws/knowledge-center/bedrock-serverless-models-access-denied). For the IAM permissions these models require, see [Bedrock setup](./bedrock-setup.md). ## Database - **MySQL 8.0+** for Co-Scientist session state and conversation history. - A dedicated schema, separate from the Seqera Platform schema. - A dedicated database host is **recommended**. Co-locating the Co-Scientist schema on the Platform's MySQL host is technically supported, but a separate host isolates resource usage, maintenance windows, and backups across Seqera products. - You will need the hostname, database name, username, and password ready for Helm configuration. ## Redis - **Redis 7.2+ or Valkey 7.2+** for caching, session state, and the automations task queue. - Redis 8.x is supported (the search/JSON/bloom modules moved into core in Redis 8.0). - Valkey 7.2+ and 8.x are supported for the default caching and task-queue workload. If you enable the optional Redis-backed knowledge index (off by default), Redis Stack 7.x or Redis 8+ is required — Valkey does not ship the `RediSearch` module. - Accessible from your cluster. - Either a dedicated instance or the instance Platform already uses. To share one instance, give Co-Scientist a different Redis database number with `agent-backend.redis.database` (Platform defaults to database `0`). - You will need the hostname and port ready for Helm configuration. ## Networking and DNS Two domains are required in addition to your Platform domain, each serving a different component: | Component | Example domain | Purpose | | --- | --- | --- | | Agent backend | `ai-api.platform.example.com` | API endpoint for the CLI and the Co-Scientist panel | | MCP server | `mcp.platform.example.com` | Model Context Protocol server | - TLS certificates for both domains. - Both domains must be subdomains of a domain shared with Platform, such as `platform.example.com`. The Co-Scientist panel authenticates to the agent backend with the Platform session cookie. `TOWER_AUTH_COOKIE_DOMAIN` scopes that cookie to the shared parent domain. The Platform Helm chart sets `TOWER_AUTH_COOKIE_DOMAIN` automatically when the agent-backend subchart is enabled. - Ingress controller configured in your cluster. ## Encryption key Generate a Fernet encryption key for encrypting sensitive tokens at rest: ```bash # using uv Python package manager (installed if not available) uv --version >/dev/null 2>&1 || curl -LsSf https://astral.sh/uv/install.sh | sh uv run --with cryptography python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())" # using Python directly, cryptography dependency module must be installed in environment python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())" ``` Store this as a Kubernetes secret. It will be referenced as `AGENT_BACKEND_TOKEN_ENCRYPTION_KEY` in the Helm values (this is the default key the `agent-backend` chart reads from `tokenEncryptionKeyExistingSecretName`). ## Kubernetes secrets Store the following values as Kubernetes secrets before installing the chart. Do not inline them in `values.yaml`. | Secret | Contains | Used by | | --- | --- | --- | | Database password | `AGENT_BACKEND_DB_PASSWORD` | Agent backend | | Redis password (if applicable) | `AGENT_BACKEND_REDIS_PASSWORD` | Agent backend | | Token encryption key | `AGENT_BACKEND_TOKEN_ENCRYPTION_KEY` | Agent backend | | Anthropic API key | `ANTHROPIC_API_KEY` (direct Anthropic path only) | Agent backend | | MCP JWT seed | `MCP_OAUTH_JWT_SECRET` 32+ char random string, `openssl rand -base64 32` | MCP server | | MCP initial access token | `MCP_OAUTH_INITIAL_ACCESS_TOKEN` (standalone MCP deploys only) | MCP server | When MCP is deployed as a subchart of the Platform parent chart, the initial access token is wired automatically from the Platform backend secret - you do not need to create it separately. When deploying MCP standalone, copy the value out of the Platform backend secret (typically named `-backend`, e.g. `platform-backend`, under the data key `OIDC_CLIENT_REGISTRATION_TOKEN`) into a new secret and reference it via `oidcToken.existingSecretName`. The MCP container loads this value as `MCP_OAUTH_INITIAL_ACCESS_TOKEN` at runtime. Bedrock authentication uses AWS IAM credentials and no API key secret is needed for the Bedrock path. On EKS, **EKS Pod Identity is the recommended approach** but IRSA or static AWS credentials on the pod are also supported. ## Local tooling - [Helm v3](https://helm.sh/docs/intro/install) - [kubectl](https://kubernetes.io/docs/tasks/tools/) - [AWS CLI v2.34.1+](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) ## Container images Co-Scientist container images are hosted at `cr.seqera.io`. The Helm charts define each image's repository path but set no default registry, so that you copy the images into your own registry. Set `global.imageRegistry` in your Platform values to that registry, or to `cr.seqera.io` if your cluster pulls directly. See the chart READMEs for the authoritative `image.registry` / `image.repository` defaults and for vendoring guidance: | Image | Source image | Chart | | --- | --- | --- | | Agent backend | `cr.seqera.io/ai/agent-backend/backend` | [agent-backend chart](https://github.com/seqeralabs/helm-charts/tree/master/charts/platform/charts/agent-backend) | | MCP server | `cr.seqera.io/enterprise/mcp/server` | [mcp chart](https://github.com/seqeralabs/helm-charts/tree/master/charts/platform/charts/mcp) | :::info From MCP 1.4.3, Seqera publishes MCP server images only to `cr.seqera.io/enterprise/mcp/server`. Earlier releases (up to 1.4.2) remain available at `cr.seqera.io/ai/mcp/server`, but Seqera publishes no new releases there. ::: Ensure your cluster can pull from `cr.seqera.io`, or if your cluster runs in a restricted network, mirror these images to your own registry. --- ## Projects Projects group the pipelines, datasets, and runs that belong to a single piece of work. Use a project to view and work with them without the noise of the rest of the workspace. A project is not a separate Platform resource. It is a **workspace label whose name starts with `proj_`**. The label is the source of truth for membership. A pipeline, dataset, or run belongs to a project when it carries the project's label. :::note Earlier releases used the `project_` prefix. Seqera Platform Enterprise 26.2 recognizes only `proj_` labels. Rename existing `project_*` labels to `proj_*` in workspace settings to keep them as projects. ::: ## Enable projects Projects are enabled by default in every organization workspace. They do not depend on the Co-Scientist agent backend. To restrict projects to specific workspaces, set `TOWER_SCIENTIST_VIEW_ALLOWED_WORKSPACES` to a comma-separated list of workspace IDs, or list the IDs in `tower.yml`: ```yaml tower: scientist-view: allowed-workspaces: - 1234567890 - 9876543210 ``` To turn projects off in every workspace, set `TOWER_SCIENTIST_VIEW_ALLOWED_WORKSPACES` to `0`. See [Co-Scientist configuration](../enterprise/configuration/overview#co-scientist). Projects are never available in personal workspaces. Restart Seqera Platform after changing either setting. ## Permissions Viewing projects requires the `project_view:read` permission. Every predefined workspace role includes it. Custom roles include it only when an organization owner selects it in the **Projects** permission category. Existing custom roles do not gain it when you upgrade to 26.2. The predefined **Project** workspace role grants access to projects without the broader workspace resource permissions. Users with this role see only the **Projects** view and cannot create or rename projects. Creating a project requires permission to create labels, and editing a project requires permission to update labels. ## Open projects When projects are enabled for a workspace and your role includes `project_view:read`, the side navigation shows a **Workspace**/**Projects** switcher. Select **Projects** to open the projects list. A role with `project_view:read` but without `workspace_resources:read`, such as the **Project** role, shows only the **Projects** view, with no switcher. If the workspace has no projects yet, the page shows a **Get started with projects** empty state with an **Add project** button. Otherwise, the list shows one row per project, plus a **\ overview** row that covers every resource in the workspace. Use **Search projects** to filter the list. Select a project to open its details page, which has **Runs**, **Datasets**, and **Reports** tabs filtered to the project's label. You can launch and relaunch pipelines from inside a project. ## Create a project 1. Select **Projects** in the side navigation. 1. Select **Add project**. 1. Enter a **Project name**. Names can contain only letters, numbers, dashes, and underscores, and must be unique in the workspace, ignoring case. 1. Optional: In the **Resources** card, select **Add** to choose the pipelines and datasets that belong to the project. 1. Select **Add**. Seqera Platform creates a workspace label named `proj_` and applies it to the resources you selected. You can also create a project from workspace settings. Go to **Labels**, create a label with the `proj_` prefix, and apply it to the pipelines and datasets that belong to the project. For example: - `proj_rnaseq` - `proj_variant_calling` - `proj_chip_seq` :::tip Create the label in workspace settings **before** applying it to resources. This ensures the label has a Platform-assigned ID, which Seqera Platform needs to auto-attach the label when you upload new datasets into the project. ::: ## Display names Seqera Platform strips the `proj_` prefix to produce the display name: | Platform label | Display name in the projects list | Browser tab title | | --- | --- | --- | | `proj_rnaseq` | rnaseq | Project rnaseq | | `proj_wgs` | wgs | Project wgs | | `proj_single_cell` | single_cell | Project single_cell | Choose descriptive names after the prefix so projects are easy to identify. ## Add resources to a project Because membership lives on the label, adding a resource to a project is the same action as applying the project's label to it: - **Pipelines and datasets**: Select them when you create or edit the project, or apply the `proj_*` label in Seqera Platform. - **Datasets uploaded inside a project**: Seqera Platform attaches the project's label automatically. - **Runs**: Runs launched from inside a project carry the project's label. The pipeline and dataset forms hide `proj_*` labels from their **Labels** field and refuse to create one there. Manage project membership from the project itself. ## Edit or delete a project Open the project's actions menu in the projects list: - **Edit**: Rename the project or change which pipelines and datasets belong to it, then select **Save**. Renaming a project renames its label. - **Delete**: Remove the project's label and its resource associations. Deleting a project does not delete the pipelines, datasets, and runs themselves. You cannot edit or delete the **\ overview** row. ## Projects and Co-Scientist You can trigger background agents from a project's **Runs** tab. The Co-Scientist panel works with the page you are viewing and the current workspace. It does not have a separate project selector. See [Co-Scientist in Seqera Platform](./platform.md). For problems with project labels and empty states, see [Co-Scientist troubleshooting](../troubleshooting_and_faqs/coscientist_troubleshooting.md). ## Learn more - [Seqera Platform labels](../labels/overview.md): Create and manage workspace labels - [Co-Scientist in Seqera Platform](./platform.md): Use the Co-Scientist panel in Seqera Platform - [Custom roles](../orgs-and-teams/custom-roles.md): Configure role permissions --- ## Quickstart This page walks you through your first Co-Scientist session: log in, start a session, switch between build mode and plan mode, debug a Platform run with a built-in skill, and set a long-running goal. :::info[**Prerequisites**]{#prerequisites} You will need the following to get started: - [Seqera CLI](./installation.mdx) - A user account on your Seqera Platform Enterprise deployment - `SEQERA_AI_BACKEND_URL` set to your organization's agent backend (see [Installation](./installation.mdx#configure-the-co-scientist-backend)) ::: ## Step 1: Log in to Seqera Platform Authenticate the CLI against your Seqera Platform account: ```bash seqera login ``` This will: 1. Open your default browser to the Seqera login page. 1. Prompt you to sign in with your Seqera Platform credentials. 1. Automatically capture the authentication token. 1. Display a success message in your terminal: ```console [Login] Starting Seqera CLI authentication... [Login] ✓ Authentication successful! [Login] ✓ Organization set: ``` :::tip See [Authentication](./authentication.md) for more information about how to log in and out, authenticate in automated environments, and manage your organization. ::: ## Step 2: Start an interactive session Launch an interactive Co-Scientist session: ```bash seqera ai ``` The Co-Scientist prompt appears, with a footer showing the active mode (**build** by default). See [Modes](./modes.md) for more information. ## Step 3: List commands and skills Show the built-in commands and available skills: ```text /help ``` :::tip Type `/` to open command autocomplete. ::: ## Step 4: Switch between build and plan modes Co-Scientist runs in two modes that control what it can do: - **Build mode** (default): Executes commands, edits files, and launches workflows - **Plan mode**: Read-only analysis and planning, for exploring options before making changes Press `Shift+Tab` to switch between modes. The active mode appears in the composer footer, and `/status` prints a full readout. Try plan mode with a comparison prompt: ```text Compare whether I should add FastQC or fastp as the first QC step in this RNA-seq pipeline, including the workflow changes each option would require ``` ## Step 5: Debug a Seqera Platform run Run the built-in debugging skill against your most recent workspace run: ```text /debug-seqera-failed-run ``` Co-Scientist fetches your most recent workspace run, inspects logs and exit codes, and walks through likely causes and fixes. You need at least one workflow run in the workspace for this skill to find something to debug. ## Step 6: Set a long-running goal Give Co-Scientist a goal to work toward across multiple turns: ```text /goal update this pipeline for AWS Batch and add nf-tests ``` Co-Scientist works across model turns until the goal completes or the attempt limit is reached. See [Use cases](./use-cases.md) for more example prompts. ## Learn more - [Skills configuration](./skills.md): Discover, create, and install skills - [Modes](./modes.md): Build, plan, and goal modes in depth - [Use cases](./use-cases.md): Seqera CLI use cases - [Skills](./reference/skills-reference.md): Built-in skills, slash commands, and session limits --- ## CLI Reference for the `seqera` commands used with Co-Scientist. To install the CLI, see [Installation](../installation.mdx). For the slash commands available inside a session, see [Skills](./skills-reference.md). ## seqera login Authenticate the CLI against your Seqera Platform account through a browser login. ```bash seqera login ``` ## seqera logout Sign out of the current session, revoke the authentication token, and remove locally stored credentials. ```bash seqera logout ``` ## seqera ai Start an interactive Co-Scientist session. Pass an optional initial query to begin with a prompt. ```bash seqera ai [query] [options] ``` | Option | Description | | --- | --- | | `[query]` | Optional initial prompt to start the session with | | `-c`, `--continue` | Continue your most recent session in the current directory | | `-s`, `--session ` | Resume a specific session by ID | | `--approval-mode ` | Set the approval mode for local commands: `basic`, `default`, or `full` (see [Command approval](../command-approval.md)) | | `--headless` | Run non-interactively and send output to stdout | | `--show-thinking` | Include thinking messages in headless output | | `--show-tools` | Include tool calls in headless output | | `--show-tool-results` | Include tool results in headless output | | `--sub-agent` | Run as a subagent with structured JSONL output | | `--events-path ` | Write all session events to a file as JSONL | | `-v`, `--verbose` | Enable verbose logging | | `--debug` | Enable debug logging | See [Sessions](../sessions.md) for usage examples. ## seqera info Show the authenticated user, the agent backend, and Seqera Platform connection details. Use it to confirm that the CLI points at your Enterprise deployment. ```bash seqera info [-o console|json] ``` | Option | Short | Description | | --- | --- | --- | | `--output ` | `-o` | Output format: `console` (default) or `json` | ## seqera org Manage your organization selection. | Command | Description | | --- | --- | | `seqera org` | View your current organization | | `seqera org list` | List all organizations | | `seqera org switch` | Switch organization | | `seqera org clear` | Clear organization selection | ## seqera skill install Install Co-Scientist as a skill or instruction file for a coding agent. ```bash seqera skill install [options] ``` | Option | Short | Description | | --- | --- | --- | | `--local` | `-l` | Install to repo root | | `--path ` | `-p` | Install to a custom path (relative or absolute) | | `--global` | `-g` | Install to home directory | | `--detect` | `-d` | Auto-detect an existing installation and update it | ## seqera skill check Verify that an installed skill matches your current CLI version. ```bash seqera skill check [options] ``` | Option | Short | Description | | --- | --- | --- | | `--update` | `-u` | Automatically update outdated skills | | `--global` | `-g` | Check only global installations | | `--local` | `-l` | Check only local (repository) installations | ## seqera update Check for a newer CLI version and print the command to install it. ```bash seqera update ``` ## seqera --version Print the installed CLI version. ```bash seqera --version ``` ## Learn more - [Installation](../installation.mdx): Install, update, and configure the CLI - [Sessions](../sessions.md): Start, continue, resume, and exit sessions - [Coding agents](../coding-agents.md): Install Co-Scientist as a skill in your coding agent - [Skills](./skills-reference.md): Built-in skills, slash commands, and session limits - [Environment variables](./environment-variables.md): Variables for authenticating and configuring the CLI --- ## Environment variables The Seqera CLI reads the following environment variables for authentication, for connecting to your Enterprise agent backend, and for controlling CLI behavior. ## Connection and authentication | Variable | Description | | --- | --- | | SEQERA_AI_BACKEND_URL | Co-Scientist agent backend endpoint used by the CLI. Overrides `backendUrl` in the [configuration file](#configuration-file). | | SEQERA_AUTH_DOMAIN | OIDC authority base URL. The CLI fetches OpenID configuration from this URL and opens the discovered authorization endpoint in your browser. Overrides `authDomain` in the configuration file. | | SEQERA_PLATFORM_API_URL | Platform API endpoint used by `seqera info`. The CLI also uses it as the OIDC authority when neither `SEQERA_AUTH_DOMAIN` nor `authDomain` is set. | | SEQERA_AUTH_CLI_CLIENT_ID | OAuth client ID for the Seqera CLI. Defaults to `seqera_ai_cli`. | | SEQERA_AUTH_AUDIENCE | OAuth audience requested during login. Defaults to `platform`. | | SEQERA_AUTH_REDIRECT_PORT | Local port for the browser login callback. Defaults to `53682`. | | SEQERA_AUTH_CALLBACK_TIMEOUT_MS | How long the CLI waits for the browser login callback, in milliseconds. | | SEQERA_BROWSER_AUTO_OPEN | Set to `false` to print the login URL instead of opening a browser. Useful on remote hosts. | | SEQERA_ACCESS_TOKEN | Platform access token for non-interactive use. When set, the CLI skips the browser login flow and uses this token directly. | | TOWER_ACCESS_TOKEN | Platform personal access token used instead of browser login. | ## CLI behavior | Variable | Description | | --- | --- | | SEQERA_AI_CWD | Working directory for local operations. Defaults to the directory you start the CLI in. | | SEQERA_CLI_OUTPUT_FORMAT | Set to `json` to make JSON the default output format for `seqera info`. | | SEQERA_THEME | Force the `light` or `dark` terminal theme instead of detecting it. | | SEQERA_REDUCE_MOTION | Disable tip rotation animations. | | SEQERA_LOG_LEVEL | CLI log level. | | SEQERA_LOG_FILE | Path of the CLI diagnostic log file. Defaults to `/tmp/seqera-cli-diag.jsonl`. | | SEQERA_DISABLE_LSP_DOWNLOAD | Set to `true` to stop the CLI from downloading the Nextflow language server. See [Nextflow LSP](../nextflow-lsp.md). | ## Configuration file The CLI also reads connection settings from `~/.config/seqera-ai/config.json`. Use it instead of exporting environment variables in every shell: ```json { "authDomain": "https://platform.example.com/api", "backendUrl": "https://ai-api.platform.example.com" } ``` Environment variables take precedence over the configuration file. The CLI resolves the OIDC authority in this order: `SEQERA_AUTH_DOMAIN`, `authDomain`, `SEQERA_PLATFORM_API_URL`. It resolves the agent backend from `SEQERA_AI_BACKEND_URL`, then `backendUrl`. :::note See [Authentication](../authentication.md#connect-to-an-enterprise-backend) for OAuth and token-based setup. ::: ## Learn more - [Authentication](../authentication.md): Log in, log out, and manage tokens - [Installation](../installation.mdx): Install, update, and configure the CLI - [CLI](./cli.md): Seqera CLI commands and options - [Skills](./skills-reference.md): Built-in skills, slash commands, and session limits --- ## Reference Look up Seqera CLI commands, environment variables, and the built-in skills and slash commands available in a Co-Scientist session. ## In this section - [CLI](./cli.md): Seqera CLI commands and options - [Environment variables](./environment-variables.md): Variables for authenticating and configuring the CLI - [Skills](./skills-reference.md): Built-in skills, slash commands, and session limits --- ## Skills This page lists the slash commands and built-in skills available in a Co-Scientist session. To learn how to discover, author, and install skills, see [Skills configuration](../skills.md). ## Slash commands Co-Scientist exposes two kinds of slash command in the `/` palette. TUI commands are handled locally by the CLI to control the session itself: | Command | Description | | --- | --- | | `/help` (`?`) | Show available commands | | `/exit` (`/quit`, `/q`) | Exit the application | | `/clear` | Clear conversation history | | `/thinking` | Toggle thinking display | | `/scroll` | Toggle auto-scroll | | `/org` | Show current organization | | `/session` | Show the current session ID | | `/sessions` | Browse and switch sessions | | `/lsp` | Show LSP server status | | `/status` | Show system status | | `/credits` | Show Enterprise usage ownership and administrator contact guidance | | `/approval` | Show or set approval mode | | `/update` | Check for CLI updates and show next steps | | `/goal ` | Set a persistent goal. Co-Scientist makes up to 3 attempts. See [Modes](../modes.md) | | `/feedback` | Open feedback form | | `/help-community` | Open community help | | `/stickers` | Get Seqera stickers | The second kind, AI commands, are backed by skills and sent to the AI backend. The built-in ones are listed below. Project and user skills you author appear alongside them in `/` and `/help`, and take precedence over a built-in skill with the same name. See [Skills configuration](../skills.md). ## Built-in skills Co-Scientist includes built-in skills. Invoke a skill directly with its slash command, or describe your task and Co-Scientist loads the matching skill. :::note Built-in skills change from release to release. Treat this page as a guide rather than an inventory. Type `/` or run `/help` in your session to see the current list for your deployment. ::: ### Build a pipeline | Command | Description | | --- | --- | | `/build-nextflow-pipeline` | Plan and scaffold a new DSL2 pipeline from scripts, notebooks, papers, or informal instructions, before you write any `.nf` code | | `/create-workflow` | Assemble a pipeline from existing modules | | `/nf-pipeline-design` | Design or refactor pipeline structure, including the shape of `main.nf`, subworkflow boundaries, and channel and metadata flow | | `/nextflow-config` | Generate and explain `nextflow.config` files, including scopes, selectors, profiles, and container runtimes | | `/nextflow-schema` | Generate `nextflow_schema.json` and sample sheet schema files | | `/triage-pipeline-parameters` | Extract each meaningful setting from source material and triage it into a parameter, a pinned constant, or a profile override | | `/nextflow-output-patterns` | Apply the correct operator idioms when aggregating per-sample outputs, including `collectFile`, `join`, `groupTuple`, and channel-level null handling | | `/create-container` | Build, claim, or recommend a verified container for a tool, and verify it runs before you write a module | | `/nf-docker-scripts` | Decide whether custom scripts belong in `bin/`, in the `template` directive, or inline | | `/search-existing-modules` | Search nf-core and the wider community for an existing module before writing your own | | `/run-module` | Discover, inspect, and run a single module with `nextflow module search`, `view`, and `run` | ### Migrate and modernize | Command | Description | | --- | --- | | `/convert-jupyter-notebook` | Convert Jupyter notebooks to Nextflow processes and workflows | | `/convert-python-script` | Convert standalone Python scripts to Nextflow | | `/convert-r-script` | Convert R scripts to Nextflow, with containers and conda environments | | `/migrate-from-snakemake` | Convert Snakemake workflows to idiomatic DSL2, using a test-first migration loop | | `/nextflow-26-syntax` | Write, migrate, or debug workflows under the v2 strict-syntax parser used by Nextflow 26+ | | `/nf-v2-boolean-params` | Fix boolean parameter handling under v2 strict syntax | | `/nf-schema-migration` | Migrate from `nf-validation` to `nf-schema` v2 | | `/maintain-nf-core-pipeline` | Bring an nf-core pipeline up to date, applying template syncs and module updates before remaining code changes | | `/nf-plugin-legacy-migration` | Move a legacy plugin to the current plugin registry format | ### Test | Command | Description | | --- | --- | | `/nf-test` | Set up nf-test and write tests for pipelines, modules, and subworkflows | | `/repair-nf-test` | Diagnose and fix failing nf-tests, snapshot mismatches, and assertion errors | ### Debug and repair | Command | Description | | --- | --- | | `/debug-local-run` | Debug a local run from `.nextflow.log`, work directories, and task error logs | | `/debug-seqera-failed-run` | Debug a failed run on Seqera Platform using workflow details, failed tasks, and logs | | `/repair-workflow` | Fix or improve an existing workflow, with the right validation loop for the problem | | `/doctor` | Run an end-to-end self-test of your Co-Scientist deployment and report pass/fail per subsystem | ### Understand a pipeline or its history | Command | Description | | --- | --- | | `/nf-pipeline-structure` | Analyze how a local pipeline is organized, including processes, workflows, modules, and data flow | | `/nf-run-history` | Summarize your recent local runs, including patterns, progress, and recurring issues | | `/nextflow-history` | Inspect the local Nextflow history and cache, and correlate runs with work directories | | `/nf-data-lineage` | Trace which inputs and processes produced a result, using the lineage store | | `/nf-aggregate` | Aggregate metrics across Seqera Platform runs with the `nf-aggregate` pipeline | | `/generate-pipeline-docs` | Generate publishable pipeline documentation from the code and its observed run patterns | | `/generate-pipeline-memory` | Build private notes on a pipeline's run history and failure patterns at your organization | ### Work with Seqera Platform | Command | Description | | --- | --- | | `/launch-workflow` | Launch a pipeline on cloud or high-performance computing (HPC) infrastructure, and choose or confirm a compute environment | | `/ce-credentials-setup` | Set up compute environments and the cloud or cluster credentials they need | | `/seqera-data-links` | Add, list, update, and delete data links, including Amazon S3, Google Cloud Storage, and Azure storage | | `/seqera-mcp` | Access Seqera Platform through Model Context Protocol (MCP) tools for structured, validated operations | | `/seqerakit` | Write `seqerakit` YAML to automate Seqera Platform setup | ### Choose tools and accelerate | Command | Description | | --- | --- | | `/find-alternative-tools` | For a given analysis step, find the credible tool options with pros, cons, licensing, and a ranked recommendation | | `/enumerate-alternative-tools` | Run the same tool enumeration as `/find-alternative-tools`. The `/create-workflow` and `/nf-pipeline-design` planning flows invoke this skill | | `/parabricks` | Get version-aware guidance on NVIDIA Parabricks `pbrun` tools and assess GPU readiness | | `/genomics-workflow-acceleration` | Add optional GPU steps to an existing genomics workflow, with runtime toggles that default to off | ### Set up and extend Nextflow | Command | Description | | --- | --- | | `/install-nextflow` | Install, upgrade, or verify Nextflow and its Java prerequisites | | `/nf-plugin-development` | Create a plugin from the official template and publish it to the plugin registry | ### Review your code | Command | Description | | --- | --- | | `/simplify` | Review changed code for reuse, quality, and efficiency, then clean up the issues found | ## Payload limits To keep session payloads small, Co-Scientist caps discovered skill context at **5,000 characters**. The total session payload cap is **20,000 characters**. ## Learn more - [Installation](../installation.mdx): Install, update, and configure the CLI - [Quickstart](../quickstart.md): Run your first Co-Scientist session - [Authentication](../authentication.md): Log in, log out, and manage sessions - [Use cases](../use-cases.md): Seqera CLI use cases - [Using Co-Scientist](../configuration.md): Configure modes, sessions, skills, command approval, and more - [Coding Agents](../coding-agents.md): Install Co-Scientist as a skill in your coding agent - [Troubleshooting](../../troubleshooting_and_faqs/coscientist_troubleshooting.md): Troubleshoot common errors --- ## Sessions A session is one interactive conversation with Co-Scientist. Co-Scientist preserves your conversation history, so you can resume earlier sessions to continue your work. This page covers how to start, continue, and exit sessions, and how to run non-interactively. ## Start a session Launch an interactive session: ```bash seqera ai ``` Start with an initial query: ```bash seqera ai "list my pipelines" ``` Set the approval mode for local commands at launch: ```bash seqera ai --approval-mode full ``` See [Command approval](./command-approval.md) for the available modes. ## Continue or resume a session Continue your most recent session: ```bash seqera ai -c ``` Continue with a follow-up question: ```bash seqera ai -c "now run the pipeline with the test profile" ``` Resume a specific session by ID: ```bash seqera ai -s ``` ## Run in headless mode Run Co-Scientist in headless mode for scripting and automation. Output is sent to stdout instead of the interactive TUI. Run a query and pipe the output: ```bash seqera ai --headless "list my pipelines" ``` Include thinking messages in the output: ```bash seqera ai --headless --show-thinking "debug my pipeline" ``` Include tool calls in the output: ```bash seqera ai --headless --show-tools "list my workflows" ``` :::note Headless mode is also auto-detected when stdout is piped, for example `seqera ai "query" | grep "result"`. ::: ## Session retention and limits The Co-Scientist agent backend keeps session history for a fixed period and caps the number of sessions: | Setting | Default | | --- | --- | | CLI session history retention | 14 days | | Seqera Platform Co-Scientist conversation retention | 180 days | | Idle time before a session's in-memory state is released | 48 hours | | Sessions per user | 100 | | Sessions per workspace | 500 | After 48 hours without activity, the backend releases a session's in-memory state. The backend keeps the conversation history, and you can still resume the session until it reaches the retention limit. After the retention limit, the backend deletes the session and you can no longer resume it. Your Seqera Platform administrator can change these defaults in the agent backend configuration. ## Interrupt a response Press `Esc` to interrupt the current response. The CLI stops the run on the Co-Scientist backend as well as in your terminal. Interrupted work does not continue in the background. ## Exit a session - Type `/exit`, `/quit`, or `/q` - Press `Ctrl+C` Your conversation history is preserved, so you can resume later with `seqera ai -c`. ## Learn more - [Modes](./modes.md): Work in build, plan, and goal modes - [Command approval](./command-approval.md): Control which commands run automatically - [Quickstart](./quickstart.md): Run your first Co-Scientist session - [Skills](./reference/skills-reference.md): Built-in skills, slash commands, and session limits --- ## Claude Code The `seqera skill` command installs a skill file that enables [Claude Code](https://claude.ai/code) to use Co-Scientist as a subagent. Once installed, Claude Code can invoke Co-Scientist directly to manage workflows, build containers, query nf-core modules, and more without leaving your environment. This page covers how to install the skill into Claude Code and keep it in sync as you update the CLI. ## `seqera skill install` Use `seqera skill install` to add the Co-Scientist skill to Claude Code. Pass the Claude Code skill directory with `--path`. Relative paths resolve from the repository root. Install into the current repository: ```bash seqera skill install --path .claude/skills/seqera ``` Install for your user account, relative to your home directory: ```bash seqera skill install --global --path .claude/skills/seqera ``` Update an existing installation by detecting where it is: ```bash seqera skill install --detect ``` :::note Without `--path`, `seqera skill install` installs the Claude Code layout (`.claude/skills/seqera`), at the repository root or, with `--global`, in your home directory. It does not prompt for a location. Run it inside a git repository unless you pass `--global` or an absolute `--path`. `--path` is a directory. The CLI creates it and writes `SKILL.md` and its supporting files inside it. ::: ### Usage ```bash seqera skill install [OPTIONS] ``` ### Options `seqera skill install` accepts the following options: | Option | Short | Description | | --- | --- | --- | | `--local` | `-l` | Install to repo root | | `--path ` | `-p` | Install to a custom path (relative or absolute) | | `--global` | `-g` | Install to home directory | | `--detect` | `-d` | Auto-detect an existing installation and update it | ## `seqera skill check` The skill file is tied to the version of the CLI that created it, so it can fall out of date when you upgrade. Use `seqera skill check` to confirm your installed skill still matches your current CLI version, and update it when it doesn't. Verify that your installed skill matches your current CLI version: ```bash seqera skill check ``` Update automatically if needed: ```bash seqera skill check --update ``` ### Usage ```bash seqera skill check [OPTIONS] ``` ### Options `seqera skill check` accepts the following options: | Option | Short | Description | | --- | --- | --- | | `--update` | `-u` | Automatically update outdated skills | | `--global` | `-g` | Check only global installations | | `--local` | `-l` | Check only local (repository) installations | ## Learn more - [Installation](./installation.mdx): Install, update, and configure the CLI - [Quickstart](./quickstart.md): Run your first Co-Scientist session - [Authentication](./authentication.md): Log in, log out, and manage sessions - [Use cases](./use-cases.md): Seqera CLI use cases - [Using Co-Scientist](./configuration.md): Configure modes, sessions, skills, command approval, and more - [Coding Agents](./coding-agents.md): Install Co-Scientist as a skill in your coding agent - [Skills](./reference/skills-reference.md): Built-in skills, slash commands, and session limits --- ## Codex The `seqera skill` command installs a skill file that enables [Codex](https://openai.com/codex) to use Co-Scientist as a subagent. Once installed, Codex can invoke Co-Scientist directly to manage workflows, build containers, query nf-core modules, and more without leaving your environment. This page covers how to install the skill into Codex and keep it in sync as you update the CLI. ## `seqera skill install` Use `seqera skill install` to add the Co-Scientist skill to Codex. Pass the Codex skill directory with `--path`. Relative paths resolve from the repository root. Install into the current repository: ```bash seqera skill install --path .codex/skills/seqera ``` Install for your user account, relative to your home directory: ```bash seqera skill install --global --path .codex/skills/seqera ``` Update an existing installation by detecting where it is: ```bash seqera skill install --detect ``` :::note Without `--path`, `seqera skill install` installs the Claude Code layout (`.claude/skills/seqera`), at the repository root or, with `--global`, in your home directory. It does not prompt for a location. Run it inside a git repository unless you pass `--global` or an absolute `--path`. `--path` is a directory. The CLI creates it and writes `SKILL.md` and its supporting files inside it. ::: ### Usage ```bash seqera skill install [OPTIONS] ``` ### Options `seqera skill install` accepts the following options: | Option | Short | Description | | --- | --- | --- | | `--local` | `-l` | Install to repo root | | `--path ` | `-p` | Install to a custom path (relative or absolute) | | `--global` | `-g` | Install to home directory | | `--detect` | `-d` | Auto-detect an existing installation and update it | ## `seqera skill check` The skill file is tied to the version of the CLI that created it, so it can fall out of date when you upgrade. Use `seqera skill check` to confirm your installed skill still matches your current CLI version, and update it when it doesn't. Verify that your installed skill matches your current CLI version: ```bash seqera skill check ``` Update automatically if needed: ```bash seqera skill check --update ``` ### Usage ```bash seqera skill check [OPTIONS] ``` ### Options `seqera skill check` accepts the following options: | Option | Short | Description | | --- | --- | --- | | `--update` | `-u` | Automatically update outdated skills | | `--global` | `-g` | Check only global installations | | `--local` | `-l` | Check only local (repository) installations | ## Learn more - [Installation](./installation.mdx): Install, update, and configure the CLI - [Quickstart](./quickstart.md): Run your first Co-Scientist session - [Authentication](./authentication.md): Log in, log out, and manage sessions - [Use cases](./use-cases.md): Seqera CLI use cases - [Using Co-Scientist](./configuration.md): Configure modes, sessions, skills, command approval, and more - [Coding Agents](./coding-agents.md): Install Co-Scientist as a skill in your coding agent - [Skills](./reference/skills-reference.md): Built-in skills, slash commands, and session limits --- ## GitHub Copilot The `seqera skill` command installs a skill file that enables [GitHub Copilot](https://github.com/features/copilot) to use Co-Scientist as a subagent. Once installed, GitHub Copilot can invoke Co-Scientist directly to manage workflows, build containers, query nf-core modules, and more without leaving your environment. This page covers how to install the skill into GitHub Copilot and keep it in sync as you update the CLI. ## `seqera skill install` Use `seqera skill install` to add the Co-Scientist skill to GitHub Copilot. Pass the GitHub Copilot skill directory with `--path`. Relative paths resolve from the repository root. Install into the current repository: ```bash seqera skill install --path .github/skills/seqera ``` GitHub Copilot uses a different directory for global installs. Install for your user account into `.copilot/skills/seqera`, relative to your home directory: ```bash seqera skill install --global --path .copilot/skills/seqera ``` Update an existing installation by detecting where it is: ```bash seqera skill install --detect ``` :::note Without `--path`, `seqera skill install` installs the Claude Code layout (`.claude/skills/seqera`), at the repository root or, with `--global`, in your home directory. It does not prompt for a location. Run it inside a git repository unless you pass `--global` or an absolute `--path`. `--path` is a directory. The CLI creates it and writes `SKILL.md` and its supporting files inside it. ::: ### Usage ```bash seqera skill install [OPTIONS] ``` ### Options `seqera skill install` accepts the following options: | Option | Short | Description | | --- | --- | --- | | `--local` | `-l` | Install to repo root | | `--path ` | `-p` | Install to a custom path (relative or absolute) | | `--global` | `-g` | Install to home directory | | `--detect` | `-d` | Auto-detect an existing installation and update it | ## `seqera skill check` The skill file is tied to the version of the CLI that created it, so it can fall out of date when you upgrade. Use `seqera skill check` to confirm your installed skill still matches your current CLI version, and update it when it doesn't. Verify that your installed skill matches your current CLI version: ```bash seqera skill check ``` Update automatically if needed: ```bash seqera skill check --update ``` ### Usage ```bash seqera skill check [OPTIONS] ``` ### Options `seqera skill check` accepts the following options: | Option | Short | Description | | --- | --- | --- | | `--update` | `-u` | Automatically update outdated skills | | `--global` | `-g` | Check only global installations | | `--local` | `-l` | Check only local (repository) installations | ## Learn more - [Installation](./installation.mdx): Install, update, and configure the CLI - [Quickstart](./quickstart.md): Run your first Co-Scientist session - [Authentication](./authentication.md): Log in, log out, and manage sessions - [Use cases](./use-cases.md): Seqera CLI use cases - [Using Co-Scientist](./configuration.md): Configure modes, sessions, skills, command approval, and more - [Coding Agents](./coding-agents.md): Install Co-Scientist as a skill in your coding agent - [Skills](./reference/skills-reference.md): Built-in skills, slash commands, and session limits --- ## Other coding agents The `seqera skill` command installs a skill file that enables coding agents such as [Cursor](https://www.cursor.com/), [OpenCode](https://opencode.ai/), [Pi](https://github.com/badlogic/pi-mono), and [Windsurf](https://windsurf.com/) to use Co-Scientist as a subagent. Once installed, these agents can invoke Co-Scientist directly to manage workflows, build containers, query nf-core modules, and more without leaving your environment. This page covers the agents the CLI supports, how to install the skill into one of them, and how to keep it in sync as you update the CLI. ## Supported agents The CLI can install the skill into the following agents. Each path is relative to the repository root for a local install, or to your home directory with `--global`: | Agent | Skill directory | | --- | --- | | [Cursor](https://www.cursor.com/) | `.cursor/skills/seqera` | | [OpenCode](https://opencode.ai/) | `.opencode/skills/seqera` | | [Pi](https://github.com/badlogic/pi-mono) | `.pi/agent/skills/seqera` | | [Windsurf](https://windsurf.com/) | `.windsurf/skills/seqera` | ## `seqera skill install` Use `seqera skill install` to add the Co-Scientist skill to your coding agent. Pass the agent's skill directory from the [supported agents](#supported-agents) table with `--path`. Relative paths resolve from the repository root. Install into the current repository, for example for Cursor: ```bash seqera skill install --path .cursor/skills/seqera ``` Install for your user account, relative to your home directory: ```bash seqera skill install --global --path .cursor/skills/seqera ``` Update an existing installation by detecting where it is: ```bash seqera skill install --detect ``` :::note Without `--path`, `seqera skill install` installs the Claude Code layout (`.claude/skills/seqera`), at the repository root or, with `--global`, in your home directory. It does not prompt for a location. Run it inside a git repository unless you pass `--global` or an absolute `--path`. `--path` is a directory. The CLI creates it and writes `SKILL.md` and its supporting files inside it. ::: ### Usage ```bash seqera skill install [OPTIONS] ``` ### Options `seqera skill install` accepts the following options: | Option | Short | Description | | --- | --- | --- | | `--local` | `-l` | Install to repo root | | `--path ` | `-p` | Install to a custom path (relative or absolute) | | `--global` | `-g` | Install to home directory | | `--detect` | `-d` | Auto-detect an existing installation and update it | ## `seqera skill check` The skill file is tied to the version of the CLI that created it, so it can fall out of date when you upgrade. Use `seqera skill check` to confirm your installed skill still matches your current CLI version, and update it when it doesn't. Verify that your installed skill matches your current CLI version: ```bash seqera skill check ``` Update automatically if needed: ```bash seqera skill check --update ``` ### Usage ```bash seqera skill check [OPTIONS] ``` ### Options `seqera skill check` accepts the following options: | Option | Short | Description | | --- | --- | --- | | `--update` | `-u` | Automatically update outdated skills | | `--global` | `-g` | Check only global installations | | `--local` | `-l` | Check only local (repository) installations | ## Learn more - [Installation](./installation.mdx): Install, update, and configure the CLI - [Quickstart](./quickstart.md): Run your first Co-Scientist session - [Authentication](./authentication.md): Log in, log out, and manage sessions - [Use cases](./use-cases.md): Seqera CLI use cases - [Using Co-Scientist](./configuration.md): Configure modes, sessions, skills, command approval, and more - [Coding Agents](./coding-agents.md): Install Co-Scientist as a skill in your coding agent - [Skills](./reference/skills-reference.md): Built-in skills, slash commands, and session limits --- ## Skills configuration Skills are reusable instruction sets that extend Co-Scientist with domain-specific workflows, prompts, and operating guidance. Co-Scientist supports two skill workflows: - **Session skills**: `SKILL.md` files discovered from project and user skill directories and sent to the Co-Scientist backend as session context when you run `seqera ai` - **Agent integrations**: skill files installed by `seqera skill install` so other coding agents can invoke Co-Scientist as a subagent :::tip See [Skills](./reference/skills-reference.md) for a list of the available built-in skills and slash commands. ::: ## Use skills in the CLI When you start `seqera ai`, the CLI discovers available skills automatically. Built-in skills come from the Co-Scientist backend, and project and user skills come from your [discovery directories](#discovery-directories). All of them appear as slash commands in the `/` command palette and `/help`. If a project or user skill has the same name as a built-in skill, your skill takes precedence. You can: - Type `/` to browse built-in commands, built-in skills, and your own skills - Run `/help` to see commands and skill descriptions in the terminal - Add project-specific `SKILL.md` files so Co-Scientist starts each session with the right context ## Skill format Each skill lives in its own directory and includes a `SKILL.md` file with YAML frontmatter: ```text my-skill/ SKILL.md references/ ``` ```markdown --- name: my-skill description: Short description of what this skill does --- Detailed instructions, examples, and guidelines. ``` `name` and `description` are required. Skills missing either field are skipped. ## Discovery directories Co-Scientist searches these directories in order. The first directory to register a skill name takes precedence, and later skills with the same name are ignored. | Priority | Path | Scope | | --- | --- | --- | | 1 | `.seqera/skills/` in the current directory and each parent directory up to the repository root, nearest first | project | | 2 | `.agents/skills/` at the same levels, checked after `.seqera/skills/` at each level | project | | 3 | `$XDG_CONFIG_HOME/seqera/skills/` (default `~/.config/seqera/skills/`) | user | | 4 | `~/.seqera/skills/` | user | | 5 | `~/.agents/skills/` | user | At each directory level, Co-Scientist checks `.seqera/skills/` before `.agents/skills/`. Outside a git repository, it checks only the current directory. Project skills take priority over user skills. A repository-specific version of a skill overrides the global one. ### Cross-agent compatibility `.agents/skills/` follows the [Agent Skills](https://agentskills.io) convention, which makes skills portable across coding agents. `.seqera/skills/` is Seqera-specific. ## Install skills into Co-Scientist You can add skills by creating the directory structure manually or by installing them from the [Agent Skills](https://agentskills.io) ecosystem: ```bash npx skills add https://github.com/vercel-labs/agent-skills --skill vercel-react-best-practices ``` After adding a skill, restart `seqera ai` so the new skill is loaded into the session. ## Install Co-Scientist into coding agents Co-Scientist can install itself as a skill or instruction file so another coding agent can invoke it as a subagent. See [Coding agents](./coding-agents.md) for the supported agents and the `seqera skill install` and `seqera skill check` commands. ## Learn more - [Installation](./installation.mdx): Install, update, and configure the CLI - [Quickstart](./quickstart.md): Run your first Co-Scientist session - [Authentication](./authentication.md): Log in, log out, and manage sessions - [Use cases](./use-cases.md): Seqera CLI use cases - [Using Co-Scientist](./configuration.md): Configure modes, sessions, skills, command approval, and more - [Coding Agents](./coding-agents.md): Install Co-Scientist as a skill in your coding agent - [Skills](./reference/skills-reference.md): Built-in skills, slash commands, and session limits - [Troubleshooting](../troubleshooting_and_faqs/coscientist_troubleshooting.md): Troubleshoot common errors --- ## Usage and cost Co-Scientist in Seqera Platform Enterprise runs in your self-hosted environment and uses your configured inference provider for Claude model access. Enterprise deployments do not use Seqera Cloud credit balances or the Cloud credit request flow. Instead, your organization manages inference access, limits, and costs through AWS Bedrock or Anthropic API. AWS Bedrock is recommended for Enterprise deployments. ## What users see In Enterprise deployments, Co-Scientist does not enforce Seqera Cloud credit balances, and the Co-Scientist usage overview shown in Seqera Platform Cloud is not available. The same applies in the [Co-Scientist panel](./platform.md) and in the CLI. If your session is blocked because of usage limits, contact your Seqera Platform administrator. The administrator can verify the agent backend configuration and inference provider account. ## What administrators manage Administrators should manage: - AWS Bedrock model access, inference profiles, quotas, and IAM roles, when Bedrock is used. - Anthropic API keys, usage limits, and billing, when direct Anthropic API access is used. - Amazon Titan embedding access, if improved documentation search is enabled. - Any organization-specific policies for Co-Scientist availability. For deployment configuration, see [Co-Scientist](../enterprise/install-seqera-coscientist.mdx). ## Learn more - [Co-Scientist](./index.md): Co-Scientist overview - [Authentication](./authentication.md): Log in, log out, and session management - [Use cases](./use-cases.md): Seqera CLI use cases - [Troubleshooting](../troubleshooting_and_faqs/coscientist_troubleshooting.md): Troubleshoot common errors --- ## Use cases Co-Scientist is an AI assistant for building, running, and managing bioinformatics workflows, available through the Seqera CLI. The sections below walk through common tasks with example prompts you can adapt to your own work: - [Develop and debug Nextflow pipelines](#develop-and-debug-nextflow-pipelines): Understand pipeline structure, generate config and schema files, debug runs, and convert scripts to Nextflow. - [Run pipelines on Seqera Platform](#run-pipelines-on-seqera-platform): Launch, monitor, and debug workflow runs in your workspace. - [Build containers with Wave](#build-containers-with-wave): Create containers from conda or pip packages without writing a Dockerfile. - [Work with data](#work-with-data): Browse data links, move files, and find reference datasets. - [Discover and run nf-core modules](#discover-and-run-nf-core-modules): Search over 1,000 nf-core modules and generate ready-to-run commands. - [Edit local project files](#edit-local-project-files): Make AI-assisted edits to files in your working directory. ## Develop and debug Nextflow pipelines Co-Scientist helps you develop, debug, and understand Nextflow pipelines with AI-powered analysis and code generation. The examples below are prompts you can adapt to your own pipeline. ### Understand your pipeline structure ```text > Show me the structure of main.nf ``` ```text > What processes are defined in this pipeline? ``` ```text > /nf-pipeline-structure ``` ### Generate configuration files ```text > /nextflow-config ``` ### Debug your pipeline ```text > /debug-local-run ``` ```text > Why is my pipeline failing? ``` ### Review local execution history ```text > /nf-run-history ``` Trace output provenance with data lineage: ```text > /nf-data-lineage ``` ### Generate schema files ```text > /nextflow-schema ``` ### Convert scripts to Nextflow ```text > /convert-python-script ``` ### Fix strict syntax ```text > /nextflow-26-syntax ``` ### Migrate old schema definitions ```text > /nf-schema-migration ``` ## Run pipelines on Seqera Platform Use Seqera Platform capabilities to run and manage workflows at scale with AI assistance. The examples below are prompts you can adapt to your own workspace. ### List your workflows ```text > List my recent workflows ``` ### Launch a pipeline ```text > Launch the nf-core/rnaseq pipeline with the test profile ``` ### Debug failed runs ```text > Why did my last workflow fail? ``` ```text > Get the logs for the failed task in my last run ``` ### Debug your most recent run ```text > /debug-seqera-failed-run ``` ## Build containers with Wave Co-Scientist can create containerized environments using Wave, without the need to write Dockerfiles. The examples below are prompts you can adapt to your own tools. ### Create a container with conda packages ```text > Create a container with samtools and bwa from bioconda ``` ### Create a container with pip packages ```text > Build a container with pandas, numpy, and scikit-learn ``` ### Get a container for a specific tool ```text > I need a container with FastQC version 0.12.1 ``` :::note Co-Scientist generates a Wave container URL that you can use directly in your Nextflow pipelines or pull with Docker. ::: ## Work with data Co-Scientist helps you manage data through Platform data links and access reference datasets. The examples below are prompts you can adapt to your own data. ### Browse data links ```text > List my data links ``` ```text > Show me the contents of my S3 data link ``` ### Download and upload files ```text > Generate a download URL for results/final_report.html ``` ```text > Upload my local results to the data link ``` ### Access reference data ```text > Find the human reference genome GRCh38 ``` ```text > Search for RNA-Seq test data ``` ## Discover and run nf-core modules Co-Scientist provides access to over 1,000 nf-core modules for common bioinformatics tasks. The examples below are prompts you can adapt to your own analysis. ### Search for modules ```text > Find nf-core modules for sequence alignment ``` ```text > What modules are available for variant calling? ``` ### Get module details ```text > Show me how to use the nf-core/bwa/mem module ``` ### Run a module ```text > Run FastQC on my FASTQ files ``` :::note Co-Scientist can generate the exact Nextflow command with the correct parameters for your data. ::: ## Edit local project files Co-Scientist can interact with files in your current working directory. The examples below are prompts you can adapt to your own project. ### Start from your project folder ```bash cd /path/to/your/project seqera ai ``` ### Ask for help with local tasks ```text > Show me the structure of main.nf ``` ```text > Add a new process to handle quality control ``` :::note Local file operations are controlled by [approval modes](./command-approval.md#approval-modes). By default, Co-Scientist asks for your approval before making changes outside your working directory or running potentially dangerous commands. ::: ## Learn more - [Modes](./modes.md): Work in build, plan, and goal modes - [Skills configuration](./skills.md): Discover, create, and install skills - [Skills](./reference/skills-reference.md): Built-in skills, slash commands, and session limits - [Command approval](./command-approval.md): Control which commands run automatically - [Code intelligence](./nextflow-lsp.md): Language-server support for Nextflow, Python, and R - [Projects](./projects.md): Organize workspace resources into projects using Platform labels - [Troubleshooting](../troubleshooting_and_faqs/coscientist_troubleshooting.md): Troubleshoot common errors --- ## AWS Batch :::tip This guide assumes you have an existing [Amazon Web Service (AWS)](https://aws.amazon.com/) account. The AWS Batch service quota for job queues is 50 per account. For more information on AWS Batch service quotas, see [AWS Batch service quotas](https://docs.aws.amazon.com/batch/latest/userguide/service_limits.html). ::: There are two ways to create a Seqera Platform compute environment for AWS Batch: - [**Automatically**](#automatic-configuration-of-batch-resources): this option lets Seqera automatically create the required AWS Batch resources in your AWS account, using an internal tool within Seqera Platform called "Forge". This removes the need to set up your AWS Batch infrastructure manually. Resources can also be automatically deleted when the compute environment is removed from Platform. - [**Manually**](#manual-configuration-of-batch-resources): this option lets Seqera use existing AWS Batch resources previously created. Both options require specific IAM permissions to function correctly, as well as access to an S3 bucket or EFS/FSx file system to store intermediate Nextflow files. ## S3 bucket creation AWS S3 (Simple Storage Service) is a type of **object storage**. To access input and output files using Seqera products like [Studios](../studios/overview) and [Data Explorer](../data/data-explorer) create one or more **S3 buckets**. An S3 bucket can also be used to store intermediate results of your Nextflow pipelines, as an alternative to using EFS or FSx file systems. :::note Using EFS or FSx as work directory is incompatible with Studios. ::: 1. Navigate to the [AWS S3 console](https://console.aws.amazon.com/s3/home). 1. In the top right of the page, select the same region where you plan to create your AWS Batch compute environment. 1. Select **Create bucket**. 1. Enter a unique name for your bucket. 1. Leave the rest of the options as default and select **Create bucket**. :::note S3 can be used by Nextflow for the storage of intermediate files. In production pipelines, this can amount to a lot of data. To reduce costs, consider using a retention policy when creating a bucket, such as automatically deleting intermediate files after 30 days. See the [AWS documentation](https://aws.amazon.com/premiumsupport/knowledge-center/s3-empty-bucket-lifecycle-rule/) for more information. ::: ## EFS or FSx file system creation [AWS Elastic File System (EFS)](https://aws.amazon.com/efs/) and [AWS FSx for Lustre](https://aws.amazon.com/fsx/lustre/) are types of **file storage** that can be used as a Nextflow work directory to store intermediate files, as an alternative to using S3 buckets. :::note Using EFS or FSx as work directory is incompatible with Studios. ::: To use EFS or FSx as your Nextflow work directory, create an EFS or FSx file system in the same region where you plan to create your AWS Batch compute environment. The creation of an EFS or FSx file system can be done automatically by Seqera when creating the AWS Batch compute environment, or manually by following the steps below. If you let Seqera create the file system automatically, it will also be deleted when the compute environment is removed from Platform, unless the "Dispose Resources" option is disabled in the advanced options. ### Creating an EFS file system To create a new EFS file system manually, visit the [EFS console](https://console.aws.amazon.com/efs/home). 1. Select **Create file system**. 1. Optionally give it a name, then select the VPC where your AWS Batch compute environment will be created. 1. Leave the rest of the options as default and select **Create file system**. ### Creating an FSx file system To create a new FSx for Lustre file system manually, visit the [FSx console](https://console.aws.amazon.com/fsx/home). 1. Select **Create file system**. 1. Select FSx for Lustre 1. Follow the prompts to configure the file system according to your requirements, then select **Next**. 1. Review the configuration and select **Create file system**. Make sure the [Lustre client](https://docs.aws.amazon.com/fsx/latest/LustreGuide/install-lustre-client.html) is available in the AMIs used by your AWS Batch compute environment to allow mounting FSx file systems. ## Networking A Studio session on an AWS Batch compute environment runs as the head job, on the head queue, in the subnets configured for the underlying Batch compute environment. Studios is not supported on a compute environment with **Enable Fargate for head job** selected. - **Studios sessions are outbound-only**: The Seqera Connect client inside a Studio session opens a tunnel outward to the Connect server, and all session traffic — including SSH when enabled — travels over that outbound connection. **No inbound path to the session is required**, so you do not need inbound security group rules or source-IP allow-lists for dynamically launched Studio jobs. The Batch compute environment subnets do need outbound access to the Connect server, through a NAT gateway or equivalent for private subnets. See [Networking](../studios/overview#networking) in the Studios documentation. For the ports and directions to configure on your firewall, see [Firewall configuration](../enterprise/advanced-topics/firewall-configuration). ## Required Platform IAM permissions To create and launch pipelines, explore buckets with Data Explorer or run Studio sessions with the AWS Batch compute environment, an IAM user with specific permissions must be provided. Some permissions are mandatory for the compute environment to be created and function correctly, while others are optional and used for example to provide list of values to pick from in the Platform UI. Permissions can be attached directly to an [IAM user](#iam-user-creation), or to an [IAM role](#iam-role-creation-optional) that the IAM user can assume when accessing AWS resources. A permissive and broad policy with all the required permissions is provided here for a quick start. However, we recommend following the principle of least privilege and only granting the necessary permissions for your use case, as shown in the following sections.
Full permissive policy (for reference) ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "BatchEnvironmentManagementCanBeRestricted", "Effect": "Allow", "Action": [ "batch:CreateComputeEnvironment", "batch:CreateJobQueue", "batch:DeleteComputeEnvironment", "batch:DeleteJobQueue", "batch:TagResource", "batch:UpdateComputeEnvironment", "batch:UpdateJobQueue" ], "Resource": [ "arn:aws:batch:*:*:compute-environment/TowerForge-*", "arn:aws:batch:*:*:job-queue/TowerForge-*" ] }, { "Sid": "BatchEnvironmentListing", "Effect": "Allow", "Action": [ "batch:DescribeComputeEnvironments", "batch:DescribeJobDefinitions", "batch:DescribeJobQueues", "batch:DescribeJobs" ], "Resource": "*" }, { "Sid": "BatchJobsManagementCanBeRestricted", "Effect": "Allow", "Action": [ "batch:CancelJob", "batch:RegisterJobDefinition", "batch:SubmitJob", "batch:TagResource", "batch:TerminateJob" ], "Resource": [ "arn:aws:batch:*:*:job-definition/*", "arn:aws:batch:*:*:job-queue/TowerForge-*", "arn:aws:batch:*:*:job/*" ] }, { "Sid": "LaunchTemplateManagement", "Effect": "Allow", "Action": [ "ec2:CreateLaunchTemplate", "ec2:DeleteLaunchTemplate", "ec2:DescribeLaunchTemplates", "ec2:DescribeLaunchTemplateVersions" ], "Resource": "*" }, { "Sid": "PassRolesToBatchCanBeRestricted", "Effect": "Allow", "Action": "iam:PassRole", "Resource": "*", "Condition": { "StringEquals": { "iam:PassedToService": [ "batch.amazonaws.com", "ec2.amazonaws.com" ] } } }, { "Sid": "CloudWatchLogsAccessCanBeRestricted", "Effect": "Allow", "Action": [ "logs:Describe*", "logs:FilterLogEvents", "logs:Get*", "logs:List*", "logs:StartQuery", "logs:StopQuery", "logs:TestMetricFilter" ], "Resource": "*" }, { "Sid": "OptionalS3PlatformDataAccessCanBeRestricted", "Effect": "Allow", "Action": [ "s3:Get*", "s3:List*", "s3:PutObject" ], "Resource": "*" }, { "Sid": "OptionalIAMManagementCanBeRestricted", "Effect": "Allow", "Action": [ "iam:AddRoleToInstanceProfile", "iam:AttachRolePolicy", "iam:CreateInstanceProfile", "iam:CreateRole", "iam:DeleteInstanceProfile", "iam:DeleteRole", "iam:DeleteRolePolicy", "iam:DetachRolePolicy", "iam:GetRole", "iam:ListAttachedRolePolicies", "iam:ListRolePolicies", "iam:PutRolePolicy", "iam:RemoveRoleFromInstanceProfile", "iam:TagInstanceProfile", "iam:TagRole" ], "Resource": [ "arn:aws:iam::*:role/TowerForge-*", "arn:aws:iam::*:instance-profile/TowerForge-*" ] }, { "Sid": "OptionalFetchOptimizedAMIMetadata", "Effect": "Allow", "Action": "ssm:GetParameters", "Resource": "arn:aws:ssm:*:*:parameter/aws/service/ecs/*" }, { "Sid": "OptionalEC2MetadataDescribe", "Effect": "Allow", "Action": [ "ec2:DescribeAccountAttributes", "ec2:DescribeImages", "ec2:DescribeInstanceTypeOfferings", "ec2:DescribeInstanceTypes", "ec2:DescribeKeyPairs", "ec2:DescribeSecurityGroups", "ec2:DescribeSubnets", "ec2:DescribeVpcs" ], "Resource": "*" }, { "Sid": "OptionalFSXManagementCanBeRestricted", "Effect": "Allow", "Action": [ "fsx:CreateFileSystem", "fsx:DeleteFileSystem", "fsx:DescribeFileSystems", "fsx:TagResource" ], "Resource": "*" }, { "Sid": "OptionalEFSManagementCanBeRestricted", "Effect": "Allow", "Action": [ "elasticfilesystem:CreateFileSystem", "elasticfilesystem:DeleteFileSystem", "elasticfilesystem:CreateMountTarget", "elasticfilesystem:DeleteMountTarget", "elasticfilesystem:DescribeFileSystems", "elasticfilesystem:DescribeMountTargets", "elasticfilesystem:UpdateFileSystem", "elasticfilesystem:PutLifecycleConfiguration", "elasticfilesystem:TagResource" ], "Resource": "*" }, { "Sid": "OptionalPipelineSecretsListing", "Effect": "Allow", "Action": "secretsmanager:ListSecrets", "Resource": "*" }, { "Sid": "OptionalPipelineSecretsManagementCanBeRestricted", "Effect": "Allow", "Action": [ "secretsmanager:DescribeSecret", "secretsmanager:DeleteSecret", "secretsmanager:CreateSecret" ], "Resource": "arn:aws:secretsmanager:*:*:secret:tower-*" }, { "Sid": "OptionalLineageIntegrationSNSAndS3", "Effect": "Allow", "Action": [ "sns:CreateTopic", "sns:SetTopicAttributes", "sns:Subscribe", "sns:ConfirmSubscription", "sns:Unsubscribe", "sns:DeleteTopic", "s3:CreateBucket", "s3:GetBucketNotification", "s3:PutBucketNotification", "s3:GetBucketLocation", "s3:ListBucket", "s3:GetObject" ], "Resource": [ "arn:aws:sns:*:*:seqera-lineage-*", "arn:aws:s3:::seqera-lineage-*", "arn:aws:s3:::seqera-lineage-*/*" ] } ] } ```
### AWS Batch management The first section of the policy allows Seqera to create, update and delete Batch compute environments ("CE"), job queues ("JQ") and jobs. If you are required to use manually created CEs and JQs or prefer to manage their lifecycle yourself, you can remove the permissions to manipulate CEs and JQs from the policy. The minimum permissions required are: - `batch:DescribeJobs` to report job status - `batch:DescribeJobDefinitions` to list existing job definitions - `batch:RegisterJobDefinition` to create new job definitions - `batch:CancelJob` to cancel jobs - `batch:SubmitJob` to submit jobs - `batch:TagResource` to tag jobs - `batch:TerminateJob` to terminate jobs You can use `batch:DescribeJobQueues` to list the existing job queues in a drop-down but it's not required if you're specifying manually created job queues. However, it is required when you let Seqera create and manage job queues automatically (using the Forge tool). In this case, the `batch:DescribeComputeEnvironments` permission must also be added. You can also restrict permissions based on resource tags. These are defined by users when they [set up a pipeline in Platform](https://docs.seqera.io/platform-enterprise/resource-labels/overview). ```json { "Sid": "BatchEnvironmentListing", "Effect": "Allow", "Action": [ "batch:DescribeJobDefinitions", "batch:DescribeJobs" ], "Resource": "*" }, { "Sid": "BatchJobsManagement", "Effect": "Allow", "Action": [ "batch:CancelJob", "batch:RegisterJobDefinition", "batch:SubmitJob", "batch:TagResource", "batch:TerminateJob" ], "Resource": [ "arn:aws:batch:::job-queue/MyCustomJQ", "arn:aws:batch:::job-definition/*", "arn:aws:batch:::job/*" ], "Condition": { "StringEqualsIfExists": { "aws:ResourceTag/MyCustomTag": "MyCustomValue" } } } ``` :::warning Restricting the `batch` actions using resource tags requires that you set the appropriate tags on each Seqera pipeline when configuring it in Platform. Forgetting to set the tag will cause the pipeline to fail to run. ::: The job definition and job name resources cannot be restricted to specific names, as Seqera creates job definitions and jobs with dynamic names. Therefore, the wildcard `*` must be used in the name of these resources. In addition, `batch:SubmitJob` requires permission on both job definitions and job queues, so make sure to include both ARNs in the `Resource` array. If you prefer to let Seqera manage Batch resources for you, you can still restrict the permissions to specific resources in your account ID and region; you can also restrict permissions based on Resource tag, as shown with the `Condition`s in the example above. :::note The quick start policy is expecting CE and JQ names automatically created by Seqera to start with the `TowerForge-` prefix, which is the default prefix used by Platform Enterprise. If you [customized it on your Enterprise installations](../enterprise/configuration/overview#compute-environments) with `TOWER_FORGE_PREFIX` adapt the policy to the new prefix. ::: ### Launch template management Seqera requires the ability to create and manage EC2 launch templates using optimized AMIs identified via AWS Systems Manager (SSM). :::note AWS does not support restricting IAM permissions on EC2 launch templates based on specific resource names or tags. As a result, permission to operate on any resource `*` must be granted. ::: ### Pass role to Batch The `iam:PassRole` permission allows Seqera to pass [execution IAM roles](https://docs.aws.amazon.com/batch/latest/userguide/execution-IAM-role.html#create-execution-role) to AWS Batch to run Nextflow pipelines. Permissions can be restricted to only allow passing the manually created roles or the roles created by Seqera automatically with the default prefix `TowerForge-` to the AWS Batch and EC2 services, in a specific account: ```json { "Sid": "PassRolesToBatch", "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam:::role/TowerForge-*", "Condition": { "StringEquals": { "iam:PassedToService": [ "batch.amazonaws.com", "ec2.amazonaws.com" ] } } } ``` ### CloudWatch logs access Seqera requires access to CloudWatch logs to display relevant log data in the web interface. The policy can be scoped down to limit access to the [specific log group](#advanced-options) defined on the compute environment in a specific account and region: ```json { "Sid": "CloudWatchLogsAccess", "Effect": "Allow", "Action": [ "logs:Describe*", "logs:FilterLogEvents", "logs:Get*", "logs:List*", "logs:StartQuery", "logs:StopQuery", "logs:TestMetricFilter" ], "Resource": "arn:aws:logs:::log-group:/aws/batch/job/*" } ``` ### S3 access (optional) Seqera automatically attempts to fetch a list of S3 buckets available in the AWS account connected to Platform, to provide them in a drop-down to be used as Nextflow working directory, and make the compute environment creation smoother. This feature is optional, and users can type the bucket name manually when setting up a compute environment. To allow Seqera to fetch the list of buckets in the account, the `s3:ListAllMyBuckets` action can be added, and it must have the `Resource` field set to `*`, as shown in the generic policy at the beginning of this document. The `s3:ListAllMyBuckets` action also allows Data Explorer to auto-discover the data repositories accessible to your workspace credentials. Seqera offers several products to manipulate data on AWS S3 buckets, such as [Studios](../studios/overview) and [Data Explorer](../data/data-explorer); if these features are not used the related permissions can be omitted. The IAM policy can be scoped down to only allow limited read/write permissions in certain S3 buckets used by Studios/Data Explorer. For each bucket you want to browse, upload to, or download from with Data Explorer, grant `s3:GetObject` and `s3:PutObject` on the bucket objects, and `s3:ListBucket`, `s3:GetBucketLocation`, `s3:GetBucketPolicy`, and `s3:GetBucketAcl` on the bucket itself. In addition, the policy must include permission to check the region and list the content of the S3 bucket used as Nextflow work directory. We also recommend granting the `s3:GetObject` permission on the work directory path to fetch Nextflow log files. :::note If you opted to create a separate S3 bucket only for Nextflow work directories, there is no need for the IAM user to have read/write access to it. If Seqera is allowed to manage resources (using Batch Forge) the IAM roles automatically created will have the necessary permissions. If you set up the compute environment manually, you can create the required IAM roles with the necessary permissions as detailed in the [manual AWS Batch setup documentation](../enterprise/advanced-topics/manual-aws-batch-setup). ::: ```json { "Sid": "S3CheckBucketWorkDirectory", "Effect": "Allow", "Action": [ "s3:GetBucketLocation", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::example-bucket-used-as-work-directory" ] }, { "Sid": "S3ReadOnlyNextflowLogFiles", "Effect": "Allow", "Action": [ "s3:GetObject" ], "Resource": [ "arn:aws:s3:::example-bucket-used-as-work-directory/path/to/work/directory/*" ] }, { "Sid": "S3ReadWriteBucketsForStudiosDataExplorer", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectTagging", "s3:GetBucketLocation", "s3:GetBucketPolicy", "s3:GetBucketAcl", "s3:ListBucket", "s3:PutObject", "s3:PutObjectTagging", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::example-bucket-read-write-studios", "arn:aws:s3:::example-bucket-read-write-studios/*", "arn:aws:s3:::example-bucket-read-write-data-explorer", "arn:aws:s3:::example-bucket-read-write-data-explorer/*" ] } ``` :::note `s3:GetBucketLocation` allows Data Explorer to resolve each bucket's region. `s3:GetBucketPolicy` and `s3:GetBucketAcl` allow it to inspect each bucket's access configuration when it lists and connects to data repositories. If you prefer not to enumerate individual actions, the `s3:Get*` and `s3:List*` wildcards shown in the full permissive policy above also cover these actions. ::: ### IAM roles for AWS Batch (optional) Seqera can automatically create the IAM roles needed to interact with AWS Batch and other AWS services. You can opt out of this behavior by creating the required IAM roles manually and providing their ARNs during compute environment creation in Platform: refer to the [documentation](../enterprise/advanced-topics/manual-aws-batch-setup) for more details on how to manually set up IAM roles. To allow Seqera to create IAM roles but restrict it to your specific account and the default IAM role prefix, use the following statement: ```json { "Sid": "IAMRoleAndProfileManagement", "Effect": "Allow", "Action": [ "iam:AddRoleToInstanceProfile", "iam:AttachRolePolicy", "iam:CreateInstanceProfile", "iam:CreateRole", "iam:DeleteInstanceProfile", "iam:DeleteRole", "iam:DeleteRolePolicy", "iam:DetachRolePolicy", "iam:GetRole", "iam:ListAttachedRolePolicies", "iam:ListRolePolicies", "iam:PutRolePolicy", "iam:RemoveRoleFromInstanceProfile", "iam:TagInstanceProfile", "iam:TagRole" ], "Resource": [ "arn:aws:iam:::role/TowerForge-*" "arn:aws:iam:::instance-profile/TowerForge-*" ] } ``` :::note The quick start policy is expecting role names automatically created by Seqera to start with the `TowerForge-` prefix, which is the default prefix used by Platform Enterprise. If you [customized it on your Enterprise installations](../enterprise/configuration/overview#compute-environments) with `TOWER_FORGE_PREFIX` adapt the policy to the new prefix. ::: ### AWS Systems Manager (optional) Seqera Platform can interact with AWS Systems Manager (SSM) to [identify ECS Optimized AMIs](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/retrieve-ecs-optimized_AMI.html) for pipeline execution. This permission is optional, meaning that a [custom AMI ID](#advanced-options) can be provided at compute environment creation, removing the need for this permission. ### EC2 describe permissions (optional) Seqera can interact with EC2 to retrieve information about existing AWS resources in your account, including VPCs, subnets, and security groups. This data is used to populate drop-downs in the Platform UI when creating new compute environments. While these permissions are optional, they are recommended to enhance the user experience. Without these permissions, resource ARNs need to be manually entered in the interface by the user. :::note AWS does not support restricting IAM permissions on EC2 Describe actions based on specific resource names or tags. As a result, permission to operate on any resource `*` must be granted. ::: ### FSx file systems (optional) Seqera can manage [AWS FSx file systems](https://aws.amazon.com/fsx/), if needed by the pipelines. This section of the policy is optional and can be omitted if FSx file systems are not used by your pipelines. The describe actions cannot be restricted to specific resources, so permission to operate on any resource `*` must be granted. The management actions can be restricted to specific resources, like in the example below. ```json { "Sid": "FSxDescribe", "Effect": "Allow", "Action": [ "fsx:DescribeFileSystems" ], "Resource": "*" }, { "Sid": "FSxManagement", "Effect": "Allow", "Action": [ "fsx:CreateFileSystem", "fsx:DeleteFileSystem", "fsx:TagResource" ], "Resource": "arn:aws:fsx:::file-system/MyManualFSx" } ``` ### EFS file systems (optional) Seqera can manage [AWS EFS file systems](https://aws.amazon.com/efs/), if needed by the pipelines. This section of the policy is optional and can be omitted if EFS file systems are not used by your pipelines. The describe actions cannot be restricted to specific resources, so permission to operate on any resource `*` must be granted. The management actions can be restricted to specific resources, like in the example below. ```json { "Sid": "EFSDescribe", "Effect": "Allow", "Action": [ "elasticfilesystem:DescribeFileSystems", "elasticfilesystem:DescribeMountTargets" ], "Resource": "*" }, { "Sid": "EFSManagement", "Effect": "Allow", "Action": [ "elasticfilesystem:CreateFileSystem", "elasticfilesystem:DeleteFileSystem", "elasticfilesystem:CreateMountTarget", "elasticfilesystem:DeleteMountTarget", "elasticfilesystem:UpdateFileSystem", "elasticfilesystem:PutLifecycleConfiguration", "elasticfilesystem:TagResource" ], "Resource": "arn:aws:elasticfilesystem:::file-system/MyManualEFS" } ``` ### Pipeline secrets (optional) Seqera can synchronize [pipeline secrets](../secrets/overview) defined on the Platform workspace with AWS Secrets Manager, which requires additional permissions on the IAM user. If you do not plan to use pipeline secrets, you can omit this section of the policy. The listing of secrets cannot be restricted, but the management actions can be restricted to only allow managing secrets in a specific account and region, which must be the same region where the pipeline runs. Note that Seqera only creates secrets with the `tower-` prefix. ```json { "Sid": "PipelineSecretsListing", "Effect": "Allow", "Action": "secretsmanager:ListSecrets", "Resource": "*" }, { "Sid": "PipelineSecretsManagementCanBeRestricted", "Effect": "Allow", "Action": [ "secretsmanager:DescribeSecret", "secretsmanager:DeleteSecret", "secretsmanager:CreateSecret" ], "Resource": "arn:aws:secretsmanager:::secret:tower-*" } ``` You can specify a customer-managed KMS key (CMK) in the **Pipeline secrets KMS key** field under **Advanced options**, or as the `TOWER_AWS_SECRETS_KMS_KEY_ID` installation default. If you do, the compute environment credentials also require `kms:GenerateDataKey` and `kms:Decrypt` on that key. Grant these actions either by naming the compute environment principal in the key policy, or in the principal's own IAM policy if the key policy delegates to IAM. The default key policy created by `aws kms create-key` delegates to IAM. ```json { "Sid": "PipelineSecretsKmsKey", "Effect": "Allow", "Action": [ "kms:GenerateDataKey", "kms:Decrypt" ], "Resource": "arn:aws:kms:::key/" } ``` #### Additional steps required to use secrets in a pipeline To successfully use pipeline secrets, the IAM roles manually created must follow the steps detailed in the [documentation](../secrets/overview#aws-secrets-manager-integration). ### Data lineage (optional) If you enable [data lineage](../data/data-lineage) in your workspace, add the following permissions to your Platform integration credentials so they can create the notification topic and bucket notifications that the lineage service uses: ```json { "Sid": "LineageIntegrationSNS", "Effect": "Allow", "Action": [ "sns:CreateTopic", "sns:SetTopicAttributes", "sns:Subscribe", "sns:ConfirmSubscription", "sns:Unsubscribe", "sns:DeleteTopic" ], "Resource": "arn:aws:sns:::seqera-lineage-*" }, { "Sid": "LineageIntegrationS3", "Effect": "Allow", "Action": [ "s3:CreateBucket", "s3:GetBucketNotification", "s3:PutBucketNotification", "s3:GetBucketLocation", "s3:ListBucket", "s3:GetObject" ], "Resource": [ "arn:aws:s3:::seqera-lineage-*", "arn:aws:s3:::seqera-lineage-*/*" ] } ``` These permissions cover **Automatic** provisioning. For **Manual** provisioning, Platform makes no control-plane calls other than confirming its own webhook subscription. See [Data lineage](../data/data-lineage#additional-iam-permissions-required) for the reduced permission set. ## Create the IAM policy The policy above must be created in the AWS account where the AWS Batch resources need to be created. 1. Open the [AWS IAM console](https://console.aws.amazon.com/iam) in the account where you want to create the AWS Batch resources. 1. From the left navigation menu, select **Policies** under **Access management**. 1. Select **Create policy**. 1. On the **Policy editor** section, select the **JSON** tab. 1. Following the instructions detailed in the [IAM permissions breakdown section](#required-platform-iam-permissions) replace the default text in the policy editor area under the **JSON** tab with a policy adapted to your use case, then select **Next**. 1. Enter a name and description for the policy on the **Review and create** page, then select **Create policy**. ## IAM user creation Seqera requires an Identity and Access Management (IAM) User to create and manage AWS Batch resources in your AWS account. We recommend creating a separate IAM policy rather an IAM User inline policy, as the latter only allows 2048 characters, which may not be sufficient for all the required permissions. In certain scenarios, for example when multiple users need to access the same AWS account and provision AWS Batch resources, an IAM role with the required permissions can be created instead, and the IAM user can assume that role when accessing AWS resources, as detailed in the [IAM role creation (optional)](#iam-role-creation-optional) section. Depending whether you choose to let Seqera automatically create the required AWS Batch resources in your account, or prefer to set them up manually, the IAM user must have specific permissions as detailed in the [Required Platform IAM permissions](#required-platform-iam-permissions) section. Alternatively, you can create an IAM role with the required permissions and allow the IAM user to assume that role when accessing AWS resources, as detailed in the [IAM role creation (optional)](#iam-role-creation-optional) section. ### Create an IAM user 1. From the [AWS IAM console](https://console.aws.amazon.com/iam), select **Users** in the left navigation menu, then select **Create User** at the top right of the page. 1. Enter a name for your user (e.g., _seqera_) and select **Next**. 1. Under **Permission options**, select **Attach policies directly**, then search for and select the policy created above, and select **Next**. * If you prefer to make the IAM user assume a role to manage AWS resources (see the [IAM role creation (optional)](#iam-role-creation-optional) section), create a policy with the following content (edit the AWS principal with the ARN of the role created) and attach it to the IAM user: ```json { "Sid": "AssumeRoleToManageBatchResources", "Effect": "Allow", "Action": "sts:AssumeRole", "Resource": "arn:aws:iam:::role/", "Condition": { "StringEquals": { "sts:ExternalId": "" } } } ``` 1. On the last page, review the user details and select **Create user**. The user has now been created. The most up-to-date instructions for creating an IAM user can be found in the [AWS documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html). ### Obtain IAM user credentials To get the credentials needed to connect Seqera to your AWS account, follow these steps: 1. From the [AWS IAM console](https://console.aws.amazon.com/iam), select **Users** in the left navigation menu, then select the newly created user from the users table. 1. Select the **Security credentials** tab, then select **Create access key** under the **Access keys** section. 1. In the **Use case** dialog that appears, select **Command line interface (CLI)**, then tick the confirmation checkbox at the bottom to acknowledge that you want to proceed creating an access key, and select **Next**. 1. Optionally provide a description for the access key, like the reason for creating it, then select **Create access key**. 1. Save the **Access key** and **Secret access key** in a secure location as you will need to provide them when creating credentials in Seqera. ## IAM role creation (optional) Rather than attaching permissions directly to the IAM user, you can create an IAM role with the required permissions and allow the IAM user to assume that role when accessing AWS resources. This is useful when multiple IAM users are used to access the same AWS account: this way the actual permissions to operate on the resources are only granted to a single centralized role. 1. From the [AWS IAM console](https://console.aws.amazon.com/iam), select **Roles** in the left navigation menu, then select **Create role** at the top right of the page. 1. Select **Custom trust policy** as the type of trusted entity, provide the following policy and edit the AWS principal with the ARN of the IAM user created in the [IAM user creation](#iam-user-creation) section, then select **Next**. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam:::user/" ] }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } }, { "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam:::user/" ] }, "Action": "sts:TagSession" } ] } ``` 1. On the **Permissions** page, search for and select the policy created in the [IAM user creation](#iam-user-creation) section, then select **Next**. 1. Give the role a name and optionally a description, review the details of the role, optionally provide tags to help you identify the role, then select **Create role**. Multiple users can be specified in the trust policy by adding more ARNs to the `Principal` section. :::note Seqera Platform generates the `External ID` value during AWS credential creation. For role-based credentials, use this exact value in your IAM trust policy (`sts:ExternalId`). ::: ### Role-based trust policy example (Seqera Enterprise) For role-based AWS credentials in Enterprise, use the AWS IAM role configured in your deployment (``) in your trust policy and enforce the `External ID` generated during credential creation: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } }, { "Effect": "Allow", "Principal": { "AWS": "" }, "Action": "sts:TagSession" } ] } ``` :::info In Seqera Enterprise, a jump role is optional. If you configure one, use your own jump role ARN as the trusted principal in the trust policy. The **Assume role** value in the credential form is the customer IAM role ARN in your AWS account. It is separate from any optional jump role configuration. ::: :::info To use role-based access with no External ID, set `TOWER_ALLOW_INSTANCE_CREDENTIALS=true` in your deployment [configuration](../enterprise/configuration/overview#compute-environments). Then create AWS credentials using an IAM role ARN only (no access key, secret key, or External ID), and remove the entire `Condition` block for `sts:ExternalId` from your trust policy. ::: ## AWS credential options AWS credentials can be configured in three ways: - **Key-based credentials**: Access key and secret key with direct IAM permissions. If you provide a role ARN in **Assume role**, the **Generate External ID** switch is displayed and External ID generation is optional. - **Role-based credentials (recommended)**: Use role assumption only (no static keys). Paste the IAM role ARN which Seqera must use for accessing your AWS resources in **Assume role**. External ID is generated automatically when you save. - **Workload identity federation**: No stored credential. Platform exchanges a short-lived OpenID Connect (OIDC) token for temporary credentials against an IAM role you control, and stores only the role Amazon Resource Name (ARN). See [Workload identity federation][wif] for the trust policy, permission policies, and setup steps. Use the IAM role ARN which Seqera must use for accessing your AWS resources in **Assume role**. This field is available for both key-based and role-based credentials. It is optional for key-based credentials and required for role-based credentials. Existing credentials created before March 2026 continue to work without changes. `TOWER_ALLOW_INSTANCE_CREDENTIALS=true` configuration behavior remains unchanged. ## Automatic configuration of Batch resources :::caution AWS Batch creates resources that you may be charged for in your AWS account. See [Cloud costs](../monitoring/cloud-costs) for guidelines to manage cloud resources effectively and prevent unexpected costs. ::: ### AWS Batch Seqera automates the configuration of an [AWS Batch](https://aws.amazon.com/batch/) compute environment and the queues required to deploy Nextflow pipelines. After your IAM user and S3 bucket have been set up, create a new **AWS Batch** compute environment in Seqera. #### Create a Seqera AWS Batch compute environment Seqera will create the head and compute [job queues](https://docs.aws.amazon.com/batch/latest/userguide/job_queues.html) and their respective [compute environments](https://docs.aws.amazon.com/batch/latest/userguide/compute_environments.html) where jobs will be executed. The job queues are configured with [job state limit actions](https://docs.aws.amazon.com/batch/latest/APIReference/API_JobStateTimeLimitAction.html) to automatically purge jobs that cannot be scheduled on any node type available for the compute environment. Depending on the provided configuration in the UI, Seqera might also create IAM roles for Nextflow head job execution, CloudWatch log groups, EFS or FSx filesystems, etc. 1. Select **Compute environments** from the navigation menu of the Seqera Workspace where you want to setup the CE. 1. Select **Add compute environment**. 1. Enter a descriptive name for this environment, e.g., _AWS Batch Spot (eu-west-1)_. 1. Select **AWS Batch** as the target platform. 1. From the **Credentials** drop-down, select existing AWS credentials, or select **+** to add new credentials. If you're using existing credentials, skip to step 9. :::note You can create multiple credentials in your Seqera environment. See [Credentials](../credentials/overview). ::: 1. Enter a name, e.g., _AWS Credentials_. 1. Under **AWS credential mode**, select **Keys** or **Role**. 1. For **Keys** mode: - Add the **Access key** and **Secret key** you [previously obtained](#obtain-iam-user-credentials). - Optionally paste the IAM role ARN which Seqera must use for accessing your AWS resources in **Assume role**. - If you paste a role ARN in **Assume role**, the **Generate External ID** switch is displayed. Generating an External ID is optional in **Keys** mode. - If **Generate External ID** is selected, an External ID is automatically generated and shown after you save the credential. 1. For **Role** mode: - Paste the IAM role ARN which Seqera must use for accessing your AWS resources in **Assume role**. - External ID is generated automatically when you save the credential. :::note When using AWS keys without an assumed role, the associated AWS user must have been granted permissions to operate on the cloud resources directly. When an assumed role is provided, the IAM user keys are only used to retrieve temporary credentials impersonating the role specified: this could be useful when e.g. multiple IAM users are used to access the same AWS account, and the actual permissions to operate on the resources are only granted to the role. ::: 1. Select a **Region**, e.g., _eu-west-1 - Europe (Ireland)_. This region must match the location of the S3 bucket or EFS/FSx file system you plan to use as work directory. 1. In the **Pipeline work directory** field type or select from the drop-down the S3 bucket [previously created](#s3-bucket-creation), e.g., `s3://seqera-bucket`. The work directory can be customized to specify a folder inside the bucket where Nextflow intermediate files will be stored, e.g., `s3://seqera-bucket/nextflow-workdir`. The bucket must be located in the same region chosen in the previous step. :::note When you specify an S3 bucket as your work directory, this bucket is used for the Nextflow [cloud cache](https://docs.seqera.io/nextflow/cache-and-resume#cache-stores) by default. Seqera adds a `cloudcache` block to the Nextflow configuration file for all runs executed with this compute environment. This block includes the path to a `cloudcache` folder in your work directory, e.g., `s3://seqera-bucket/cloudcache/.cache`. You can specify an alternative cache location with the **Nextflow config file** field on the pipeline [launch](../launch/launchpad#launch-form) form. ::: Similarly you can specify a path in an EFS or FSx file system as your work directory. When using EFS or FSx, you'll need to scroll down to "EFS file system" or "FSx for Lustre" sections to specify either an existing file system ID or let Seqera create a new one for you automatically. Read the notes in steps 23 and 24 below on how to setup EFS or FSx. :::warning Using an EFS or FSx file system as your work directory is currently incompatible with [Studios](../studios/overview), and will result in errors with checkpoints and mounted data. Use an S3 bucket as your work directory when using Studios. ::: 1. Select **Enable Wave containers** to facilitate access to private container repositories and provision containers in your pipelines using the Wave containers service. See [Wave containers](https://docs.seqera.io/nextflow/wave) for more information. 1. Select **Enable Fusion v2** to allow access to your S3-hosted data via the [Fusion v2](https://docs.seqera.io/fusion) virtual distributed file system. This speeds up most data operations. The Fusion v2 file system requires Wave containers to be enabled. See [Fusion file system](../supported_software/fusion/overview) for configuration details.
Use Fusion v2 file system :::note The compute recommendations below are based on internal benchmarking performed by Seqera. Benchmark runs of [nf-core/rnaseq](https://github.com/nf-core/rnaseq) used profile `test_full`, consisting of an input dataset with 16 FASTQ files and a total size of approximately 123.5 GB. ::: We recommend using Fusion with AWS NVMe instances (fast instance storage) as this delivers the fastest performance when compared to environments using only AWS EBS (Elastic Block Store). 1. Use Seqera Platform version 23.1 or later. 1. Use an S3 bucket as the pipeline work directory. 1. Enable **Wave containers**, **Fusion v2**, and **fast instance storage**. 1. Select the **Batch Forge** config mode. 1. Fast instance storage requires an EC2 instance type that uses NVMe disks. Specify NVMe-based instance types in **Instance types** under **Advanced options**. If left unspecified, Platform selects instances from AWS NVMe-based instance type families. See [Instance store temporary block storage for EC2 instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html) for more information. :::note When enabling fast instance storage, do not select the `optimal` instance type families (c4, m4, r4) for your compute environment as these are not NVMe-based instances. Specify AWS NVMe-based instance types, or leave the **Instance types** field empty for Platform to select NVMe instances for you. ::: :::tip We recommend selecting 8xlarge or above for large and long-lived production pipelines: - A local temp storage disk of at least 200 GB and a random read speed of 1000 MBps or more. To work with files larger than 100 GB, increase temp storage accordingly (400 GB or more). - Dedicated networking ensures a guaranteed network speed service level compared with "burstable" instances. See [Instance network bandwidth](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-network-bandwidth.html) for more information. ::: When using Fusion v2 without fast instance storage, the following EBS settings are applied to optimize file system performance: - EBS boot disk size is increased to 100 GB - EBS boot disk type GP3 is selected - EBS boot disk throughput is increased to 325 MB/s Extensive benchmarking of Fusion v2 has demonstrated that the increased cost associated with these settings are generally outweighed by the costs saved due to decreased run time.
1. Select **Enable Fusion Snapshots (beta)** to enable Fusion to automatically restore jobs that are interrupted when an AWS Spot instance reclamation occurs. Requires Fusion v2 and a **Spot** provisioning model. See [Fusion Snapshots](https://docs.seqera.io/fusion/guide/snapshots) for more information. :::caution Restrict **Instance types** under **Advanced options** to the [recommended instance types](https://docs.seqera.io/fusion/guide/snapshots/aws#selecting-an-ec2-instance). Enabling Fusion Snapshots does not populate this field. Do not enable Fusion Snapshots on an On-Demand compute environment. ::: 1. Set the **Config mode** to **Batch Forge** to allow Seqera Platform to manage AWS Batch compute environments using the Forge tool. 1. Select a **Provisioning model**. To minimize compute costs select **Spot**. You can specify an allocation strategy and instance types under [**Advanced options**](#advanced-options). If advanced options are omitted, Seqera Platform 23.2 and later versions default to `BEST_FIT_PROGRESSIVE` for On-Demand and `SPOT_PRICE_CAPACITY_OPTIMIZED` for Spot compute environments. :::note You can create a compute environment that launches either Spot or On-Demand instances. Spot instances can cost as little as 20% of On-Demand instances, and with Nextflow's ability to automatically relaunch failed tasks, Spot is almost always the recommended provisioning model. Note, however, that when choosing Spot instances, Seqera will also create a dedicated queue for running the main Nextflow job using a single On-Demand instance to prevent any execution interruptions. From Nextflow version 24.10, the default Spot reclamation retry setting changed to `0` on AWS and Google. By default, no internal retries are attempted on these platforms. Spot reclamations now lead to an immediate failure, exposed to Nextflow in the same way as other generic failures (for example, returning `exit code 1` on AWS). Nextflow treats these failures like any other job failure unless you configure a retry strategy. For more information, see [Spot instance failures and retries](../troubleshooting_and_faqs/nextflow#spot-instance-failures-and-retries). ::: 1. Enter the **Max CPUs**, e.g., `64`. This is the maximum number of combined CPUs (the sum of all instances' CPUs) AWS Batch will provision at any time. 1. Select **EBS Auto scale (deprecated)** to allow the EC2 virtual machines to dynamically expand the amount of available disk space during task execution. This feature is deprecated, and is not compatible with Fusion v2. :::note When you run large AWS Batch clusters (hundreds of compute nodes or more), EC2 API rate limits may cause the deletion of unattached EBS volumes to fail. You should delete volumes that remain active after Nextflow jobs have completed to avoid additional costs. Monitor your AWS account for any orphaned EBS volumes via the EC2 console, or with a Lambda function. See [here](https://aws.amazon.com/blogs/mt/controlling-your-aws-costs-by-deleting-unused-amazon-ebs-volumes/) for more information. ::: 1. With the optional **Enable Fusion mounts (deprecated)** feature enabled, S3 buckets specified in **Pipeline work directory** and **Allowed S3 Buckets** are mounted as file system volumes in the EC2 instances carrying out the Batch job execution. These buckets can then be accessed at `/fusion/s3/`. For example, if the bucket name is `s3://imputation-gp2`, your pipeline will access it using the file system path `/fusion/s3/imputation-gp2`. **Note:** This feature has been deprecated. Consider using Fusion v2 (see above) for enhanced performance and stability. :::note You do not need to modify your pipeline or files to take advantage of this feature. Nextflow will automatically recognize and replace any reference to files prefixed with `s3://` with the corresponding Fusion mount paths. ::: 1. Select **Enable Fargate for head job** to run the Nextflow head job with the [AWS Fargate](https://aws.amazon.com/fargate/) container service and speed up pipeline launch. Fargate is a serverless compute engine that enables users to run containers without the need to provision servers or clusters in advance. AWS takes a few minutes to spin up an EC2 instance, whereas jobs can be launched with Fargate in under a minute (depending on container size). We recommend Fargate for most pipeline deployments, but EC2 is more suitable for environments that use GPU instances, custom AMIs, or that require more than 16 vCPUs. If you specify a custom AMI ID in the [Advanced options](#advanced-options) below, this will not be applied to the Fargate-enabled head job. See [here](https://docs.aws.amazon.com/batch/latest/userguide/fargate.html#when-to-use-fargate) for more information on Fargate's limitations. :::note Fargate requires the Fusion v2 file system and a **Spot** provisioning model. Fargate is not compatible with EFS and FSx file systems. ::: 1. Select **Enable GPUs** if you intend to run GPU-dependent workflows in the compute environment. See [GPU usage](./overview#aws-batch) for more information. :::note Seqera only supports NVIDIA GPUs. Select instances with NVIDIA GPUs for your GPU-dependent processes. ::: 1. Select **Use Graviton CPU architecture** to execute on Graviton-based EC2 instances (i.e., ARM64 CPU architecture). When enabled, `m6g`, `r6g`, and `c6g` instance types are used by default for compute jobs, but 3rd-generation Graviton [instances](https://www.amazonaws.cn/en/ec2/graviton/) are also supported. You can specify your own **Instance types** under [**Advanced options**](#advanced-options). :::note Graviton requires Fargate, Wave containers, and Fusion v2 file system to be enabled. This feature is not compatible with GPU-based architecture. ::: 1. Enter any additional **Allowed S3 buckets** that your workflows require to read input data or write output data. The **Pipeline work directory** bucket above is added by default to the list of **Allowed S3 buckets**. 1. To use an **EFS** file system in your pipeline, you can either select **Use existing EFS file system** and specify an existing EFS instance, or select **Create new EFS file system** to create one. To use the EFS file system as the work directory of the compute environment specify `/work` in the **Pipeline work directory** field (step 10 of this guide). - To use an existing EFS file system, enter the **EFS file system id** and **EFS mount path**. This is the path where the EFS volume is accessible to the compute environment. For simplicity, we recommend that you use `/mnt/efs` as the EFS mount path. - To create a new EFS file system, enter the **EFS mount path**. We advise that you specify `/mnt/efs` as the EFS mount path. - EFS file systems created by Batch Forge are automatically tagged in AWS with `Name=TowerForge-`, with `` being the compute environment ID. Any manually-added resource label with the key `Name` (capital N) will override the automatically-assigned `TowerForge-` label. - A custom EC2 security group needs to be configured to allow the compute environment to access the EFS file system. * Visit the [AWS Console for Security groups](https://console.aws.amazon.com/ec2/home?#SecurityGroups) and switch to the region where your workload will run. * Select **Create security group**. * Enter a relevant name like `seqera-efs-access-sg` and description, e.g., _EFS access for Seqera Batch compute environment_. * Empty both **Inbound rules** and **Outbound rules** sections by deleting default rules. * Optionally add **Tags** to the security group, then select **Create security group**. * After creating the security group, select it from the security groups list, then select the **Inbound rules** tab and select **Edit inbound rules**. * Select **Add rule** and configure the new rule as follows: - **Type**: `NFS` - **Source**: `Custom` and enter the security group ID that you're editing (you can search for it by name, e.g., `seqera-efs-access-sg`). This allows resources associated with the same security group to communicate with each other. * Select **Save rules** to finalize the inbound rule configuration. * Repeat the same steps to add an outbound rule to allow all outbound traffic: set type `All traffic` and destination `Anywhere-IPv4`/`Anywhere-IPv6`. * See the [AWS documentation about EFS security groups](https://docs.aws.amazon.com/efs/latest/ug/network-access.html) for more information. * The Security group then needs to be defined in the **Advanced options** below to allow the compute environment to access the EFS file system. :::warning EFS file systems cannot be used as work directory for [Studios](../studios/overview), but can be mounted and used by applications running in Studios. ::: 1. To use a **FSx for Lustre** file system in your pipeline, you can either select **Use existing FSx file system** and specify an existing FSx instance, or select **Create new FSx file system** to create one. To use the FSx file system as your work directory, specify `/work` in the **Pipeline work directory** field (step 10 of this guide). - To use an existing FSx file system, enter the **FSx DNS name** and **FSx mount path**. The FSx mount path is the path where the FSx volume is accessible to the compute environment. For simplicity, we recommend that you use `/mnt/fsx` as the FSx mount path. - To create a new FSx file system, enter the **FSx size** (in GB) and the **FSx mount path**. We advise that you specify `/mnt/fsx` as the FSx mount path. - FSx file systems created by Batch Forge are automatically tagged in AWS with `Name=TowerForge-`, with `` being the compute environment ID. Any manually-added resource label with the key `Name` (capital N) will override the automatically-assigned `TowerForge-` label. - A custom EC2 security group needs to be configured to allow the compute environment to access the FSx file system. * Visit the [AWS Console for Security groups](https://console.aws.amazon.com/ec2/home?#SecurityGroups) and switch to the region where your workload will run. * Select **Create security group**. * Enter a relevant name like `seqera-fsx-access-sg` and description, e.g., _FSx access for Seqera Batch compute environment_. * Empty both **Inbound rules** and **Outbound rules** sections by deleting default rules. * Optionally add **Tags** to the security group, then select **Create security group**. * After creating the security group, select it from the security groups list, then select the **Inbound rules** tab and select **Edit inbound rules**. * Select **Add rule** and configure the new rule as follows: - **Type**: `Custom TCP` - **Port range**: `988` - **Source**: `Custom` and enter the security group ID that you're editing (you can search for it by name, e.g., `seqera-fsx-access-sg`). This allows resources associated with the same security group to communicate with each other. * Repeat the step to add another rule with: - **Type**: `Custom TCP` - **Port range**: `1018-1023` - **Source**: `Custom`, same as above. * Select **Save rules** to finalize the inbound rule configuration. * Repeat the same steps to add an outbound rule to allow all outbound traffic: set type `All traffic` and destination `Anywhere-IPv4`/`Anywhere-IPv6`. * See the [AWS documentation about FSx security groups](https://docs.aws.amazon.com/fsx/latest/LustreGuide/limit-access-security-groups.html) for more information. * The Security group then needs to be defined in the **Advanced options** below to allow the compute environment to access the FSx file system. - You may need to install the `lustre` client in the AMI used by your compute environment to access FSx file systems. See [Installing the Lustre client](https://docs.aws.amazon.com/fsx/latest/LustreGuide/install-lustre-client.html) for more information. :::warning FSx file systems cannot be used as work directory for [Studios](../studios/overview), but can be mounted and used by applications running in Studios. ::: 1. Select **Dispose resources** to automatically delete all AWS resources created by Seqera Platform when you delete the compute environment, including EFS/FSx file systems. 1. Apply [**Resource labels**](../resource-labels/overview) to the cloud resources produced by this compute environment. Workspace default resource labels are prefilled. 1. Expand **Staging options** to include: - Optional [pre- or post-run Bash scripts](../launch/advanced#pre-and-post-run-scripts) that execute before or after the Nextflow pipeline execution in your environment. - Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the **Nextflow config file** field in the pipeline launch form. These values can be overridden during pipeline launch. :::info Configuration settings in this field override the same values in the pipeline repository `nextflow.config` file. See [Nextflow config file](../launch/advanced#nextflow-config-file) for more information on configuration priority. ::: 1. Under **Environment variables**, add each variable with a **Name**, **Value**, and **Target Environment**: - **Head job**: Adds the variable to the Nextflow head job container, which evaluates `nextflow.config` and submits tasks to the compute backend. Use this target for variables that Nextflow or its plugins read, such as `NXF_OPTS`, `NXF_JVM_ARGS`, `NXF_PLUGINS_DEFAULT`, or proxy settings the head node uses to reach external services. - **Compute job**: Adds the variable to the worker containers that run individual pipeline tasks. Use this target for variables your pipeline tools read, such as `OPENAI_API_KEY` for a process that calls the OpenAI API, registry credentials needed inside the task container, or tool-specific settings like `JAVA_HOME`. - **Head and Compute jobs**: Adds the variable to both the head job and the compute jobs. Use this target for values needed in both places, such as an HTTP proxy used by both Nextflow and task tools, or a credential needed in both the head job and individual compute tasks. :::note For sensitive values such as API keys and tokens, use [pipeline secrets](../secrets/overview) instead of custom environment variables. Custom environment variables are stored in the compute environment configuration and cannot be edited after creation. To rotate a value, recreate the compute environment. ::: 1. Configure any advanced options described in the next section, as needed. 1. Select **Create** to finalize the compute environment setup. It will take a few seconds for all the AWS resources to be created before you are ready to launch pipelines. :::info See [Launch pipelines](../launch/launchpad) to start executing workflows in your AWS Batch compute environment. ::: ### Advanced options Seqera Platform compute environments for AWS Batch include advanced options to configure instance types, resource allocation, custom networking, and CloudWatch and ECS agent integration. #### Seqera AWS Batch advanced options - Specify the **Allocation strategy** and indicate any preferred **Instance types**. AWS applies quotas for the number of running and requested [Spot](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-limits.html) and [On-Demand](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-on-demand-instances.html#ec2-on-demand-instances-limits) instances per account. AWS will allocate instances from up to 20 instance types, based on those requested for the compute environment. AWS excludes the largest instances when you request more than 20 instance types. :::note If these advanced options are omitted, allocation strategy defaults are `BEST_FIT_PROGRESSIVE` for On-Demand and `SPOT_PRICE_CAPACITY_OPTIMIZED` for Spot compute environments. ::: :::caution Platform CLI (known as `tw`) v0.8 and earlier do not support the `SPOT_PRICE_CAPACITY_OPTIMIZED` allocation strategy in AWS Batch. You cannot currently use CLI to create or otherwise interact with AWS Batch Spot compute environments that use this allocation strategy. ::: - Configure a custom networking setup using the **VPC ID**, **Subnets**, and **Security groups** fields. * If not defined, the default VPC, subnets, and security groups for the selected region will be used. * When using EFS or FSx file systems, select the security group previously created to allow access to the file system. The VPC ID the security group belongs to needs to match the VPC ID defined for the Seqera Batch compute environment. - You can specify a custom **AMI ID**. :::note From version 24.2, Seqera supports Amazon Linux 2023 ECS-optimized AMIs, in addition to previously supported Amazon Linux-2 AMIs. AWS-recommended Amazon Linux 2023 AMI names start with `al2023-`. To learn more about approved versions of the Amazon ECS-optimized AMIs or creating a custom AMI, see [this AWS guide](https://docs.aws.amazon.com/batch/latest/userguide/compute_resource_AMIs.html#batch-ami-spec). If a custom AMI is specified and the **Enable GPU** option is also selected, the custom AMI will be used instead of the AWS-recommended GPU-optimized AMI. ::: - If you need to debug the EC2 instance provisioned by AWS Batch, specify a **Key pair** to log in to the instance via SSH. - You can set **Min CPUs** to be greater than `0`, in which case some EC2 instances will remain active. An advantage of this is that pipeline executions will initialize faster. :::note Setting Min CPUs to a value greater than 0 will keep the required compute instances active, even when your pipelines are not running. This will result in additional AWS charges. ::: - Use **Head job CPUs** and **Head job memory** to specify the hardware resources allocated for the Nextflow head job. The default head job memory allocation is 4096 MiB. :::warning Setting head job values will also limit the size of any Studio session that can be created in the compute environment. ::: - Use **Head job role** and **Compute job role** to grant fine-grained IAM permissions to the **Head job** and **Compute jobs**. - Add an execution role ARN to the **Batch execution role** field to grant permissions to make API calls on your behalf to the ECS container used by Batch. This is required if the pipeline launched with this compute environment needs access to the secrets stored in this workspace. This field can be ignored if you are not using secrets. - Use **Pipeline secrets KMS key** to specify a customer-managed KMS key that encrypts the temporary AWS Secrets Manager secrets Seqera creates for runs that use pipeline secrets. This field accepts a key ARN or a key ID. A key ARN must be in the same region as the compute environment. Leave this field empty to use the installation default set with [`TOWER_AWS_SECRETS_KMS_KEY_ID`](../enterprise/configuration/overview#compute-environments), or the default AWS-managed key if neither is set. See [Pipeline secrets (optional)](#pipeline-secrets-optional) for the KMS permissions your compute environment credentials require. - Specify an EBS block size (in GB) in the **EBS auto-expandable block size** field to control the initial size of the EBS auto-expandable volume. New blocks of this size are added when the volume begins to run out of free space. This feature is deprecated, and is not compatible with Fusion v2. - Enter the **Boot disk size** (in GB) to specify the size of the boot disk in the VMs created by this compute environment. - If you're using **Spot** instances, you can also specify the **Cost percentage**, which is the maximum allowed price of a **Spot** instance as a percentage of the **On-Demand** price for that instance type. Spot instances will not be launched until the current Spot price is below the specified cost percentage. - Use **AWS CLI tool path** to specify the location of the `aws` CLI. - Specify a **CloudWatch Log group** for the `awslogs` driver to stream the logs entry to an existing Log group in Cloudwatch. - Specify a custom **ECS agent configuration** for the ECS agent parameters used by AWS Batch. This is appended to the `/etc/ecs/ecs.config` file in each cluster node. :::note Altering this file may result in a malfunctioning Batch Forge compute environment. See [Amazon ECS container agent configuration](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-config.html) to learn more about the available parameters. ::: ## Manual configuration of Batch resources This section is for users with a pre-configured AWS environment: follow the [AWS Batch queue and compute environment creation instructions](../enterprise/advanced-topics/manual-aws-batch-setup.mdx) to set up the required AWS Batch resources in your account. A [S3 bucket](#s3-bucket-creation) or EFS/FSx file system is required to store Nextflow intermediate files when using Seqera with AWS Batch. Refer to the [IAM user creation](#iam-user-creation) section to ensure that your IAM user has the necessary permissions to run pipelines in Seqera Platform. Remove any permissions that are not required for your use case. ### Seqera manual compute environment With your AWS environment and resources set up and your user permissions configured, create an AWS Batch compute environment in Seqera. :::caution AWS Batch creates resources that you may be charged for in your AWS account. See [Cloud costs](../monitoring/cloud-costs) for guidelines to manage cloud resources effectively and prevent unexpected costs. ::: 1. After logging in to your Seqera installation and selecting a workspace from the drop-down at the top of the page, select **Compute environments** from the navigation menu. 1. Select **Add compute environment**. 1. Enter a descriptive name for this environment, e.g., _AWS Batch Spot (eu-west-1)_. 1. Select **AWS Batch** as the target platform. 1. From the **Credentials** drop-down, select existing AWS credentials, or select **+** to add new credentials. If you're using existing credentials, skip to step 9. :::note You can create multiple credentials in your Seqera environment. See [Credentials](../credentials/overview). ::: 1. Enter a name, e.g., _AWS Credentials_. 1. Under **AWS credential mode**, select **Keys** or **Role**. 1. For **Keys** mode: - Add the **Access key** and **Secret key** you [previously obtained](#obtain-iam-user-credentials). - Optionally paste the IAM role ARN which Seqera must use for accessing your AWS resources in **Assume role**. - If you paste a role ARN in **Assume role**, the **Generate External ID** switch is displayed. Generating an External ID is optional in **Keys** mode. - If **Generate External ID** is selected, an External ID is automatically generated and shown after you save the credential. 1. For **Role** mode: - Paste the IAM role ARN which Seqera must use for accessing your AWS resources in **Assume role**. - External ID is generated automatically when you save the credential. :::note When using AWS keys without an assumed role, the associated AWS user must have been granted permissions to operate on the cloud resources directly. When an assumed role is provided, the IAM user keys are only used to retrieve temporary credentials impersonating the role specified: this could be useful when e.g. multiple IAM users are used to access the same AWS account, and the actual permissions to operate on the resources are only granted to the role. ::: 1. Select a **Region**, e.g., _eu-west-1 - Europe (Ireland)_. This region must match the region where your S3 bucket or EFS/FSx work directory is located to avoid high data transfer costs. 1. Enter or select from the drop-down the S3 bucket [previously created](#s3-bucket-creation) in the **Pipeline work directory** field, e.g., `s3://seqera-bucket`. This bucket must be in the same region chosen in the previous step to avoid incurring high data transfer costs. The work directory can be customized to specify a folder inside the bucket, e.g., `s3://seqera-bucket/nextflow-workdir`. :::note When you specify an S3 bucket as your work directory, this bucket is used for the Nextflow [cloud cache](https://docs.seqera.io/nextflow/cache-and-resume#cache-stores) by default. Seqera adds a `cloudcache` block to the Nextflow configuration file for all runs executed with this compute environment. This block includes the path to a `cloudcache` folder in your work directory, e.g., `s3://seqera-bucket/cloudcache/.cache`. You can specify an alternative cache location with the **Nextflow config file** field on the pipeline [launch](../launch/launchpad#launch-form) form. ::: Similarly you can specify a path in an EFS or FSx file system as your work directory. When using EFS or FSx, you'll need to scroll down to "EFS file system" or "FSx for Lustre" sections to specify either an existing file system ID or let Seqera create a new one for you automatically. Read the notes in steps 23 and 24 below on how to setup EFS or FSx. :::warning Using an EFS or FSx file system as your work directory is currently incompatible with [Studios](../studios/overview), and will result in errors with checkpoints and mounted data. Use an S3 bucket as your work directory when using Studios. ::: 1. Select **Enable Wave containers** to facilitate access to private container repositories and provision containers in your pipelines using the Wave containers service. See [Wave containers](https://docs.seqera.io/nextflow/wave) for more information. 1. Select **Enable Fusion v2** to allow access to your S3-hosted data via the [Fusion v2](https://docs.seqera.io/fusion) virtual distributed file system. This speeds up most data operations. The Fusion v2 file system requires Wave containers to be enabled. See [Fusion file system](../supported_software/fusion/overview) for configuration details.
Use Fusion v2 file system :::note The compute recommendations below are based on internal benchmarking performed by Seqera. Benchmark runs of [nf-core/rnaseq](https://github.com/nf-core/rnaseq) used profile `test_full`, consisting of an input dataset with 16 FASTQ files and a total size of approximately 123.5 GB. ::: We recommend using Fusion with AWS NVMe instances (fast instance storage) as this delivers the fastest performance when compared to environments using only AWS EBS (Elastic Block Store). 1. Use Seqera Platform version 23.1 or later. 1. Use an S3 bucket as the pipeline work directory. 1. Enable **Wave containers**, **Fusion v2**, and **fast instance storage**. 1. Select the **Batch Forge** config mode. 1. Fast instance storage requires an EC2 instance type that uses NVMe disks. Specify NVMe-based instance types in **Instance types** under **Advanced options**. If left unspecified, Platform selects instances from AWS NVMe-based instance type families. See [Instance store temporary block storage for EC2 instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html) for more information. :::note When enabling fast instance storage, do not select the `optimal` instance type families (c4, m4, r4) for your compute environment as these are not NVMe-based instances. Specify AWS NVMe-based instance types, or leave the **Instance types** field empty for Platform to select NVMe instances for you. ::: :::tip We recommend selecting 8xlarge or above for large and long-lived production pipelines: - A local temp storage disk of at least 200 GB and a random read speed of 1000 MBps or more. To work with files larger than 100 GB, increase temp storage accordingly (400 GB or more). - Dedicated networking ensures a guaranteed network speed service level compared with "burstable" instances. See [Instance network bandwidth](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-network-bandwidth.html) for more information. ::: When using Fusion v2 without fast instance storage, the following EBS settings are applied to optimize file system performance: - EBS boot disk size is increased to 100 GB - EBS boot disk type GP3 is selected - EBS boot disk throughput is increased to 325 MB/s Extensive benchmarking of Fusion v2 has demonstrated that the increased cost associated with these settings are generally outweighed by the costs saved due to decreased run time.
1. Select **Enable Fusion Snapshots (beta)** to enable Fusion to automatically restore jobs that are interrupted when an AWS Spot instance reclamation occurs. Requires Fusion v2 and a **Spot** provisioning model. See [Fusion Snapshots](https://docs.seqera.io/fusion/guide/snapshots) for more information. :::caution Restrict **Instance types** under **Advanced options** to the [recommended instance types](https://docs.seqera.io/fusion/guide/snapshots/aws#selecting-an-ec2-instance). Enabling Fusion Snapshots does not populate this field. Do not enable Fusion Snapshots on an On-Demand compute environment. ::: 1. Set the **Config mode** to **Manual**. 1. Enter the **Head queue** created following the [instructions](../enterprise/advanced-topics/manual-aws-batch-setup.mdx), which is the name of the AWS Batch queue that the Nextflow main job will run. 1. Enter the **Compute queue**, which is the name of the AWS Batch queue where tasks will be submitted. 1. Apply [**Resource labels**](../resource-labels/overview) to the cloud resources produced by this compute environment. Workspace default resource labels are prefilled. 1. Expand **Staging options** to include: - Optional [pre- or post-run Bash scripts](../launch/advanced#pre-and-post-run-scripts) that execute before or after the Nextflow pipeline execution in your environment. - Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the **Nextflow config file** field in the pipeline launch form. These values can be overridden during pipeline launch. :::info Configuration settings in this field override the same values in the pipeline repository `nextflow.config` file. See [Nextflow config file](../launch/advanced#nextflow-config-file) for more information on configuration priority. ::: 1. Under **Environment variables**, add each variable with a **Name**, **Value**, and **Target Environment**: - **Head job**: Adds the variable to the Nextflow head job container, which evaluates `nextflow.config` and submits tasks to the compute backend. Use this target for variables that Nextflow or its plugins read, such as `NXF_OPTS`, `NXF_JVM_ARGS`, `NXF_PLUGINS_DEFAULT`, or proxy settings the head node uses to reach external services. - **Compute job**: Adds the variable to the worker containers that run individual pipeline tasks. Use this target for variables your pipeline tools read, such as `OPENAI_API_KEY` for a process that calls the OpenAI API, registry credentials needed inside the task container, or tool-specific settings like `JAVA_HOME`. - **Head and Compute jobs**: Adds the variable to both the head job and the compute jobs. Use this target for values needed in both places, such as an HTTP proxy used by both Nextflow and task tools, or a credential needed in both the head job and individual compute tasks. :::note For sensitive values such as API keys and tokens, use [pipeline secrets](../secrets/overview) instead of custom environment variables. Custom environment variables are stored in the compute environment configuration and cannot be edited after creation. To rotate a value, recreate the compute environment. ::: 1. Configure any advanced options described in the next section, as needed. 1. Select **Create** to finalize the compute environment setup. :::info See [Launch pipelines](../launch/launchpad) to start executing workflows in your AWS Batch compute environment. ::: ### Advanced options Seqera compute environments for AWS Batch include advanced options to configure resource allocation, execution roles, custom AWS CLI tool paths, and CloudWatch integration. - Configure a custom networking setup using the **VPC ID**, **Subnets**, and **Security groups** fields. * If not defined, the default VPC, subnets, and security groups for the selected region will be used. * When using EFS or FSx file systems, select the security group previously created to allow access to the file system. The VPC ID the security group belongs to needs to match the VPC ID defined for the Seqera Batch compute environment. - Use **Head job CPUs** and **Head job memory** to specify the hardware resources allocated for the Nextflow head job. The default head job memory allocation is 4096 MiB. - Use **Head job role** and **Compute job role** to grant fine-grained IAM permissions to the head job and compute jobs, - Add an execution role ARN to the **Batch execution role** field to grant permissions to make API calls on your behalf to the ECS container used by Batch. This is required if the pipeline launched with this compute environment needs access to the secrets stored in this workspace. This field can be ignored if you are not using secrets. - Use **Pipeline secrets KMS key** to specify a customer-managed KMS key that encrypts the temporary AWS Secrets Manager secrets Seqera creates for runs that use pipeline secrets. This field accepts a key ARN or a key ID. A key ARN must be in the same region as the compute environment. Leave this field empty to use the installation default set with [`TOWER_AWS_SECRETS_KMS_KEY_ID`](../enterprise/configuration/overview#compute-environments), or the default AWS-managed key if neither is set. See [Pipeline secrets (optional)](#pipeline-secrets-optional) for the KMS permissions your compute environment credentials require. - Use **AWS CLI tool path** to specify the location of the `aws` CLI. - Specify a **CloudWatch Log group** for the `awslogs` driver to stream the logs entry to an existing Log group in Cloudwatch. :::caution Seqera is designed to terminate compute resources when a Nextflow pipeline completes or is canceled. However, due to external factors — including user-defined workflow logic, transient cloud faults, or abnormal pipeline exits — residual resources may persist. While Seqera provides visibility to detect and resolve these states, customers are responsible for final resource cleanup and ensuring compute environments operate according to Platform expectations. From Nextflow v24.10+, compute jobs are identifiable by Seqera workflow ID. If you search your AWS console/CLI/API for jobs prefixed by a given workflow ID, you can check the status and perform additional cleanup in edge case scenarios. ::: [wif]: ../credentials/workload_identity --- ## AWS Cloud :::note This compute environment type is currently in public preview. Please consult this guide for the latest information on recommended configuration and limitations. This guide assumes you already have an AWS account with a valid AWS subscription. ::: The current implementation of compute environments for cloud providers all rely on the use of batch services such as AWS Batch, Azure Batch, and Google Batch for the execution and management of submitted jobs, including pipelines and Studio session environments. Batch services are suitable for large-scale workloads, but they add management complexity. In practical terms, the currently used batch services result in some limitations: - **Long launch delay**: When you launch a pipeline or Studio in a batch compute environment, there's a delay of several minutes before the pipeline or Studio session environment is in a running state. This is caused by the batch services that need to provision the associated compute service to run a single job. - **Complex setup**: Standard batch services require complex identity management policies and configuration of multiple services, including compute environments, job queues, job definitions, etc. - **Allocation constraints**: AWS Batch and other cloud batch services have strict resource quotas. For example, a hard limit of 50 job queues per account per region. This means that no new compute environment can be created when this quota limit is reached. The AWS Cloud compute environment addresses these pain points with: - **Faster startup time**: Nextflow pipelines reach a `Running` status and Studio sessions connect in under a minute (a 4x improvement compared to classic AWS Batch compute environments). - **Simplified configuration**: Fewer configurable options, with opinionated defaults, provide the best Nextflow pipeline and Studio session execution environment, with both Wave and Fusion enabled. - **Fewer AWS dependencies**: Only one IAM role in AWS is required. IAM roles are subject to a 1000 soft limit per account. - **Spot instances**: Studios can be launched on a Spot instance. This type of compute environment is best suited to run Studios and small to medium-sized pipelines. It offers more predictable compute pricing, given the fixed instance type. It spins up a standalone EC2 instance and executes a Nextflow pipeline or Studio session with a local executor on the EC2 machine. At the end of the execution, the instance is terminated. ## Limitations - The Nextflow pipeline will run entirely on a single EC2 instance. If the instance does not have sufficient resources, the pipeline execution will fail. For this reason, the number of tasks Nextflow can execute in parallel is limited by the number of cores of the instance type selected. If you need more computing resources, you must create a new compute environment with a larger instance type. This makes the compute environment less suited for larger, more complex pipelines. ## Supported regions The following regions are currently supported: - `eu-west-1` - `us-east-1` - `us-west-2` - `eu-west-2` - `us-east-2` - `eu-central-1` - `us-west-1` - `eu-west-3` - `ap-southeast-1` ## Networking An AWS Cloud compute environment launches a single EC2 instance into a VPC subnet. Set **VPC ID**, **Subnets**, and **Security groups** under [Advanced options](#advanced-options). - **Outbound access is required**: The instance must reach Amazon S3 for the work directory, and Seqera Platform. A private subnet — one that does not auto-assign a public IP address — reaches S3 only through an S3 gateway endpoint or a NAT gateway. When Seqera Intelligent Compute selects a subnet for you, Seqera creates an S3 gateway endpoint in the VPC if one does not already exist, using the permissions described in [Additional IAM permissions](#additional-iam-permissions). If you select the subnet yourself, provide that route. - **Studios sessions are outbound-only**: The Seqera Connect client inside a Studio session opens a tunnel outward to the Connect server, and all session traffic — including SSH when enabled — travels over that outbound connection. **No inbound path to the session VM is required**, so you do not need inbound security group rules or source-IP allow-lists for dynamically launched Studio instances. The subnet does need outbound access to the Connect server. See [Networking](../studios/overview#networking) in the Studios documentation. - **Seqera does not create inbound rules**: Where Seqera manages a security group for you, it authorizes egress only. The policy grants `ec2:AuthorizeSecurityGroupEgress` and no inbound equivalent. For the ports and directions to configure on your firewall, see [Firewall configuration](../enterprise/advanced-topics/firewall-configuration). ## Requirements ### Platform credentials To create and launch pipelines or Studio sessions with this compute environment type, you must attach Seqera credentials for the cloud provider. Some permissions are mandatory for the compute environment to be created and function correctly; others are optional and used to pre-fill options in Platform. AWS credentials can be configured in three ways: - **Key-based credentials**: Access key and secret key with direct IAM permissions. If you provide a role ARN in **Assume role**, the **Generate External ID** switch is displayed and External ID generation is optional. - **Role-based credentials (recommended)**: Use role assumption only (no static keys). Paste the IAM role ARN which Seqera must use for accessing your AWS resources in **Assume role**. External ID is generated automatically when you save. - **Workload identity federation**: No stored credential. Platform exchanges a short-lived OpenID Connect (OIDC) token for temporary credentials against an IAM role you control, and stores only the role Amazon Resource Name (ARN). See [Workload identity federation][wif] for the trust policy, permission policies, and setup steps. Use the IAM role ARN which Seqera must use for accessing your AWS resources in **Assume role**. This field is available for both key-based and role-based credentials. It is optional for key-based credentials and required for role-based credentials. Existing credentials created before March 2026 continue to work without changes. `TOWER_ALLOW_INSTANCE_CREDENTIALS=true` configuration behavior remains unchanged. ### Role-based trust policy example (Seqera Enterprise) For role-based AWS credentials in Enterprise, use the AWS IAM role configured in your deployment (``) in your trust policy and enforce the `External ID` generated during credential creation: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } }, { "Effect": "Allow", "Principal": { "AWS": "" }, "Action": "sts:TagSession" } ] } ``` :::info In Seqera Enterprise, a jump role is optional. If you configure one, use your own jump role ARN as the trusted principal in the trust policy. The **Assume role** value in the credential form is the customer IAM role ARN in your AWS account. It is separate from any optional jump role configuration. ::: :::info To use role-based access with no External ID, set `TOWER_ALLOW_INSTANCE_CREDENTIALS=true` in your deployment [configuration](../enterprise/configuration/overview#compute-environments). Then create AWS credentials using an IAM role ARN only (no access key, secret key, or External ID), and remove the entire `Condition` block for `sts:ExternalId` from your trust policy. ::: ### Required permissions #### Compute environment creation The following permissions are required to provision resources in the AWS account. Only IAM roles that will be assumed by the EC2 instance must be provisioned: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "AwsCloudCreate", "Effect": "Allow", "Action": [ "iam:CreateRole", "iam:AddRoleToInstanceProfile", "iam:CreateInstanceProfile", "iam:AttachRolePolicy", "iam:PutRolePolicy", "iam:PassRole", "iam:TagRole", "iam:TagInstanceProfile" ], "Resource": "*" } ] } ``` ### Compute environment validation The following permissions are required to validate the compute environment at creation time. Seqera validates the input provided and that the resource ARNs exist in the target AWS account: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "AwsCloudValidate", "Effect": "Allow", "Action": [ "ec2:DescribeInstanceTypes", "ec2:DescribeImages", "ec2:DescribeSubnets", "ec2:DescribeSecurityGroups" ], "Resource": "*" } ] } ``` #### Pipeline and Studio session management The following permissions are required to launch pipelines, run Studio sessions, fetch live execution logs from CloudWatch, download logs from S3, and stop the execution: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "AwsCloudLaunch", "Effect": "Allow", "Action": [ "ec2:RunInstances", "ec2:DescribeInstances", "ec2:CreateTags", "ec2:TerminateInstances", "ec2:DeleteTags", "logs:GetLogEvents", "s3:GetObject" ], "Resource": "*" } ] } ``` #### Compute environment termination and resource disposal The following permissions are required to remove resources created by Seqera when the compute environment is deleted: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "AwsCloudDelete", "Effect": "Allow", "Action": [ "iam:GetRole", "iam:ListAttachedRolePolicies", "iam:ListRolePolicies", "iam:DeleteRole", "iam:DeleteInstanceProfile", "iam:RemoveRoleFromInstanceProfile", "iam:DetachRolePolicy", "iam:DeleteRolePolicy" ], "Resource": "*" } ] } ``` #### Optional permissions The following permissions enable Seqera to populate values for drop-down fields. If missing, the input fields will not be auto-populated but can still be manually entered. Though optional, these permissions are recommended for a smoother and less error-prone user experience. The `s3:ListAllMyBuckets` action also allows Data Explorer to auto-discover the data repositories accessible to your workspace credentials: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "AwsCloudRead", "Effect": "Allow", "Action": [ "ec2:DescribeInstanceTypes", "ec2:DescribeKeyPairs", "ec2:DescribeVpcs", "ec2:DescribeImages", "ec2:DescribeSubnets", "ec2:DescribeSecurityGroups", "s3:ListAllMyBuckets" ], "Resource": "*" } ] } ``` #### Data lineage (optional) If you enable [data lineage](../data/data-lineage) in your workspace, add the following permissions to your Platform integration credentials so they can create the notification topic and bucket notifications that the lineage service uses: ```json { "Sid": "LineageIntegrationSNS", "Effect": "Allow", "Action": [ "sns:CreateTopic", "sns:SetTopicAttributes", "sns:Subscribe", "sns:ConfirmSubscription", "sns:Unsubscribe", "sns:DeleteTopic" ], "Resource": "arn:aws:sns:::seqera-lineage-*" }, { "Sid": "LineageIntegrationS3", "Effect": "Allow", "Action": [ "s3:CreateBucket", "s3:GetBucketNotification", "s3:PutBucketNotification", "s3:GetBucketLocation", "s3:ListBucket", "s3:GetObject" ], "Resource": [ "arn:aws:s3:::seqera-lineage-*", "arn:aws:s3:::seqera-lineage-*/*" ] } ``` These permissions cover **Automatic** provisioning. For **Manual** provisioning, Platform makes no control-plane calls other than confirming its own webhook subscription. See [Data lineage](../data/data-lineage#additional-iam-permissions-required) for the reduced permission set. ## Seqera Intelligent Compute :::info[Private preview] Seqera Intelligent Compute is in private preview. [Contact us](https://seqera.io/intelligent-compute/) to request access. ::: Seqera Intelligent Compute is an optional capability that executes Nextflow tasks on a Seqera-managed Amazon ECS cluster instead of running them entirely on the head EC2 instance. The AWS Cloud compute environment scales beyond the resources of a single instance while preserving its fast startup behavior. When you enable Seqera Intelligent Compute, Seqera provisions and manages all ECS infrastructure on your behalf, including clusters, capacity providers, task definitions, IAM roles, and (optionally) Auto Scaling Groups for spot and on-demand capacity. All managed resources use the `seqera-sched-` prefix and are torn down automatically when no longer needed. ### Additional IAM permissions :::info[Private preview] Seqera Intelligent Compute is in private preview. [Contact us](https://seqera.io/intelligent-compute/) to request access. ::: To enable Seqera Intelligent Compute, attach an additional IAM policy (beyond the [Required permissions](#required-permissions)) to the same IAM user or role that Seqera uses to access your AWS account. The policy scopes ARN-eligible actions to the `seqera-sched-*` resource prefix, except for CloudWatch Logs actions, which are scoped to the `/seqera/*` log-group prefix (Seqera writes logs to groups such as `/seqera/platform`). The remaining `Resource: "*"` entries correspond to AWS APIs that do not support resource-level permissions, such as EC2 `Describe*`, ECR authorization tokens, and Cost Explorer.
Seqera Intelligent Compute policy ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "ECSScopedOperations", "Effect": "Allow", "Action": [ "ecs:CreateCluster", "ecs:DeleteCluster", "ecs:DescribeClusters", "ecs:PutClusterCapacityProviders", "ecs:CreateCapacityProvider", "ecs:DeleteCapacityProvider", "ecs:DescribeCapacityProviders", "ecs:RunTask", "ecs:StopTask", "ecs:DescribeTasks", "ecs:DescribeContainerInstances", "ecs:UpdateContainerInstancesState", "ecs:TagResource" ], "Resource": "arn:aws:ecs:*:*:*/seqera-sched-*" }, { "Sid": "ECSUnscopedOperations", "Effect": "Allow", "Action": [ "ecs:RegisterTaskDefinition", "ecs:DeregisterTaskDefinition", "ecs:DescribeTaskDefinition", "ecs:ListTaskDefinitions", "ecs:ListTaskDefinitionFamilies", "ecs:ListTasks" ], "Resource": "*" }, { "Sid": "IAMRoleManagement", "Effect": "Allow", "Action": [ "iam:CreateRole", "iam:GetRole", "iam:DeleteRole", "iam:PutRolePolicy", "iam:DeleteRolePolicy", "iam:ListRolePolicies", "iam:AttachRolePolicy", "iam:DetachRolePolicy", "iam:ListAttachedRolePolicies", "iam:CreateInstanceProfile", "iam:GetInstanceProfile", "iam:AddRoleToInstanceProfile", "iam:ListInstanceProfilesForRole", "iam:RemoveRoleFromInstanceProfile", "iam:DeleteInstanceProfile" ], "Resource": [ "arn:aws:iam::*:role/seqera-sched-*", "arn:aws:iam::*:instance-profile/seqera-sched-*" ] }, { "Sid": "PassRoleToECS", "Effect": "Allow", "Action": "iam:PassRole", "Resource": [ "arn:aws:iam::*:role/seqera-sched-*", "arn:aws:iam::*:role/TowerForge-*" ], "Condition": { "StringEquals": { "iam:PassedToService": [ "ecs-tasks.amazonaws.com", "ecs.amazonaws.com", "ec2.amazonaws.com" ] } } }, { "Sid": "ServiceLinkedRoles", "Effect": "Allow", "Action": "iam:CreateServiceLinkedRole", "Resource": "arn:aws:iam::*:role/aws-service-role/*", "Condition": { "StringEquals": { "iam:AWSServiceName": [ "ecs.amazonaws.com", "ecs-compute.amazonaws.com", "autoscaling.amazonaws.com", "spot.amazonaws.com" ] } } }, { "Sid": "CloudWatchLogs", "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:DeleteLogGroup", "logs:PutRetentionPolicy", "logs:DescribeLogStreams", "logs:GetLogEvents", "logs:TagResource" ], "Resource": "arn:aws:logs:*:*:log-group:/seqera/*" }, { "Sid": "EC2NetworkDiscovery", "Effect": "Allow", "Action": [ "ec2:DescribeImages", "ec2:DescribeVpcs", "ec2:DescribeSubnets", "ec2:DescribeSecurityGroups", "ec2:DescribeRouteTables", "ec2:DescribeVpcEndpoints", "ec2:DescribeInstances", "ec2:CreateSecurityGroup", "ec2:CreateVpcEndpoint", "ec2:AuthorizeSecurityGroupEgress", "ec2:CreateTags" ], "Resource": "*" }, { "Sid": "ECRAccess", "Effect": "Allow", "Action": [ "ecr:GetAuthorizationToken", "ecr:BatchCheckLayerAvailability", "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage" ], "Resource": "*" }, { "Sid": "S3Access", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:ListBucket", "s3:ListAllMyBuckets" ], "Resource": "*" }, { "Sid": "ASGEC2Operations", "Effect": "Allow", "Action": [ "ec2:DescribeInstanceTypes", "ec2:CreateLaunchTemplate", "ec2:DeleteLaunchTemplate", "ec2:RunInstances" ], "Resource": "*" }, { "Sid": "ASGManagement", "Effect": "Allow", "Action": [ "autoscaling:CreateAutoScalingGroup", "autoscaling:UpdateAutoScalingGroup", "autoscaling:DeleteAutoScalingGroup", "autoscaling:CreateOrUpdateTags" ], "Resource": "arn:aws:autoscaling:*:*:*/seqera-sched-*" }, { "Sid": "ASGDescribe", "Effect": "Allow", "Action": "autoscaling:DescribeAutoScalingGroups", "Resource": "*" }, { "Sid": "SSMECSOptimizedAmi", "Effect": "Allow", "Action": "ssm:GetParameter", "Resource": "arn:aws:ssm:*:*:parameter/aws/service/ecs/optimized-ami/*" }, { "Sid": "CostExplorer", "Effect": "Allow", "Action": "ce:GetCostAndUsage", "Resource": "*" } ] } ``` Some statements in the policy above are conditional and can be omitted depending on your deployment: - The `ASGEC2Operations` and `ASGManagement` statements are required only if you enable Auto Scaling Group-backed clusters (managed instances). Omit them for Fargate-only deployments. - The `CreateECSServiceLinkedRole` is required only if the Service Role is not already created. - The `CostExplorer` statement is only required if you enable Cost Analysis.
## Managed Amazon Machine Image (AMI) The AWS Cloud compute environment uses an AMI maintained by Seqera, and the pipeline launch procedure assumes that some basic tooling is already present in the image itself. If you want to provide your own AMI, it must include at least the following: - Docker engine, configured to run at startup. - CloudWatch agent. - The ability to shut down with the `shutdown` command. If this is missing, EC2 instances will keep running and accumulate additional costs. ## Advanced options - **Instance Type**: The EC2 instance type used by the compute environment. Choosing the instance type will directly allocate the CPU and memory available for computation. See [EC2 instance types](https://aws.amazon.com/ec2/instance-types/) for a comprehensive list of instance types and their resource limitations. - **Graviton architecture**: Enable the use of Graviton instances. AWS Graviton processors, based on the ARM64 architecture, tend to offer a better performance-to-price ratio, however, the tooling used by your pipelines must be compatible with ARM architecture. - **AMI ID**: The ID of the AMI that will be used to launch the EC2 instance. Use Seqera-maintained AMIs for best performance. - **Key pair**: The [EC2 key pair](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) to enable SSH connectivity to the running instance. If unspecified, no SSH key will be present in the running EC2 instance. - **VPC ID**: The ID of the VPC where the EC2 instance will be launched. If unspecified, the default VPC will be used. - **Subnets**: The list of VPC subnets where the EC2 instance will run. If unspecified, all the subnets of the VPC will be used. - **Security groups**: The security groups the EC2 instance will be a part of. If unspecified, no security groups will be used. - **Instance Profile**: The ARN of the `InstanceProfile` used by the EC2 instance to assume a role while running. If unspecified, Seqera will provision one with enough permissions to run. See [Custom instance profile](#custom-instance-profile) for the minimum permissions required if you provide your own. - **Pipeline secrets KMS key**: A customer-managed KMS key (CMK) that encrypts the temporary AWS Secrets Manager secrets Seqera creates for runs that use pipeline secrets. This field accepts a key ARN or a key ID. A key ARN must be in the same region as the compute environment. If unspecified, Seqera uses the installation default set with [`TOWER_AWS_SECRETS_KMS_KEY_ID`](../enterprise/configuration/overview#compute-environments), or the default AWS-managed key if neither is set. See [Custom instance profile](#custom-instance-profile) for the KMS permissions the instance profile role requires. - **Boot disk size**: The size of the EBS boot disk for the EC2 instance. If undefined, a default 50 GB `gp3` volume will be used. ### Custom instance profile When you specify a custom **Instance Profile** ARN in Advanced options, the IAM role attached to that instance profile must include the following minimum permissions. These mirror what Seqera provisions automatically when no instance profile is specified. #### Trust policy The role must be assumable by the EC2 service: ```json { "Version": "2012-10-17", "Statement": { "Effect": "Allow", "Action": "sts:AssumeRole", "Principal": { "Service": "ec2.amazonaws.com" } } } ``` #### AWS managed policies Attach the following AWS managed policies to the role: | Policy | Purpose | |--------|---------| | `arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy` | Push metrics and logs to CloudWatch | | `arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess` | Read-only access to S3 (required by Fusion and Nextflow) | | `arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly` | Pull container images from private ECR repositories | #### Inline policies In addition to the managed policies, attach the following inline policies: **S3 read/write** — grants full object access on the compute environment work directory bucket. Add one statement per bucket if you configure additional buckets under **Allow buckets**: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "ListObjectsInBucket", "Effect": "Allow", "Action": ["s3:ListBucket"], "Resource": "arn:aws:s3:::" }, { "Sid": "AllObjectActions", "Effect": "Allow", "Action": "s3:*Object", "Resource": "arn:aws:s3:::/*" }, { "Sid": "AllowObjectTagging", "Effect": "Allow", "Action": ["s3:PutObjectTagging", "s3:GetObjectTagging"], "Resource": "arn:aws:s3:::/*" } ] } ``` **Secrets Manager** — grants access to the pipeline secrets Seqera stores in AWS Secrets Manager under the `tower-` prefix. Seqera creates each referenced secret when a pipeline launches and deletes it on completion: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue", "secretsmanager:CreateSecret", "secretsmanager:DeleteSecret" ], "Resource": ["arn:aws:secretsmanager::*:secret:tower-*"] }, { "Effect": "Allow", "Action": ["secretsmanager:ListSecrets"], "Resource": ["*"] } ] } ``` If the compute environment specifies a **Pipeline secrets KMS key**, the role also requires `kms:Decrypt` on that key to read secrets encrypted with the key at task start. Grant the action either by naming the role in the key policy, or in the role's own IAM policy if the key policy delegates to IAM. The default key policy created by `aws kms create-key` delegates to IAM: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "PipelineSecretsKmsKey", "Effect": "Allow", "Action": ["kms:Decrypt"], "Resource": "arn:aws:kms:::key/" } ] } ``` **KMS for S3** — required if any of the S3 buckets used by the compute environment are encrypted with a customer-managed KMS key (SSE-KMS): ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "KmsS3Read", "Effect": "Allow", "Action": ["kms:Decrypt", "kms:DescribeKey"], "Resource": "arn:aws:kms:*:*:key/*", "Condition": { "StringLike": { "kms:ViaService": "s3.*.amazonaws.com" } } }, { "Sid": "KmsS3Write", "Effect": "Allow", "Action": ["kms:Encrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*"], "Resource": "arn:aws:kms:*:*:key/*", "Condition": { "StringLike": { "kms:ViaService": "s3.*.amazonaws.com" } } } ] } ``` :::note If your AWS account enforces EBS volume encryption at the account level (either via account default encryption settings or an SCP that requires `encrypted=true` on `RunInstances`), the EC2 instance will use a KMS key to encrypt its boot volume. In this case, the instance role must also have `kms:Decrypt`, `kms:GenerateDataKey`, `kms:CreateGrant`, and `kms:DescribeKey` permissions on the relevant KMS key — these are not included in the KMS for S3 policy above, which is scoped to S3 only. Contact your AWS administrator to identify the correct KMS key ARN and add permissions accordingly. ::: When you use a custom instance profile, note that Seqera will not create or manage the IAM role — you are responsible for keeping it up to date as requirements change. [wif]: ../credentials/workload_identity --- ## AWS Spot interruption management In AWS Batch environments that use Spot instances, tasks can be interrupted when AWS reclaims instances. This is a normal part of how Spot instances operate. The frequency of interruptions varies based on factors including the wider demand on AWS services. AWS shows the frequency of Spot reclamations in its **instance-advisor** service, which you can find [here](https://aws.amazon.com/ec2/spot/instance-advisor/). In Seqera Platform, Spot reclamations sometimes appear with logging messages like `Host EC2 (instance i-0282b396e52b4c95d) terminated` and produce non-specific exit codes such as `143` (representing `SIGTERM`) or even no exit code at all (`-`), depending on the order in which the underlying AWS components have been destroyed. If you see unexpected task failures with one or more of these features, especially with no obvious application error, review your Spot configuration and retry strategy. The following practices reduce the impact of Spot interruptions and help critical tasks retry or recover reliably. ## Recommended mitigations ### Use an On-Demand compute environment For workflows with a significant proportion of long-running processes, the costs of working with Spot, and the mitigations it requires, can outweigh the benefits. It can be simpler, and possibly cheaper, to run those workloads in On-Demand compute environments. ### Move long-running tasks to On-Demand Tasks with long runtimes are particularly vulnerable to Spot termination. In Platform, you can explicitly assign critical or long-duration tasks to On-Demand queues and leave other tasks to run in a default Spot queue: ```bash process { withName: 'run_bcl2fastq' { queue = 'TowerForge-MyOnDemandQueue' } } ``` If you don't already have one, create an On-Demand compute environment in Seqera Platform. When it's available, find the corresponding On-Demand queue name under **Compute Environments** in the Platform UI. Locate the configuration for your On-Demand environment, then scroll down to the **Manual Config Attributes** section. This section lists key configuration details, including queue names. Look for the queue name prefixed with `TowerForge-` if Forge created it. ### Use retry strategies for Spot Interruptions #### Handle retries in Nextflow by setting `errorStrategy` and `maxRetries` A generic retry strategy at the Nextflow level can be more appropriate when run times are short enough that retries are likely to succeed. Configure it as follows: ```bash process { errorStrategy = 'retry' maxRetries = 3 } ``` This example configuration applies to all types of job failure. Because Spot reclamations do not produce diagnostic exit codes, you cannot configure retries at the Nextflow level specifically for reclamations. Given the escalating costs of repeated retries, an On-Demand queue is likely more cost-effective than a very large number of retries. If you still see failures after you apply this configuration, On-Demand queues are likely more effective at limiting costs and runtimes. #### Handle retries in AWS by setting `aws.batch.maxSpotAttempts` If all processes in your workflow have runtimes short enough to complete before reclamation, consider configuring automatic retries in case of interruption: `aws.batch.maxSpotAttempts = 3` This is a global setting (not configurable per process). In this example, it lets a job retry up to three times on a new Spot instance if AWS reclaims the original instance. Retries happen automatically within AWS and restart the task from the beginning. Because this occurs within AWS, you won't see any evidence of the retries in Platform. As far as Nextflow (and Platform) is concerned, only one attempt has occurred. Nextflow submits the task again to AWS, up to any `maxRetries` configuration you have in place (see earlier). The total number of retries in that case is `maxRetries` * `aws.batch.maxSpotAttempts`. For a long-running process that is preempted repeatedly, this can represent significant costs in time and compute. :::note Starting with Nextflow version 24.08.0-edge, the default value for this setting is `0` to help avoid unexpected expenses. Be careful when you activate this setting. ::: ### Implement Spot-to-On-Demand fallback logic If you prefer to optimize for cost but ensure task reliability, consider a hybrid fallback pattern: ```bash process { withName: 'run_bcl2fastq' { errorStrategy = 'retry' maxRetries = 2 queue = { task.attempt > 1 ? 'TowerForge-MyOnDemandQueue' : 'TowerForge-MySpotQueue' } } } ``` With this setup, Nextflow sends the first attempt of a task to the Spot queue and directs any retries to the On-Demand queue, where they won't be preempted. This helps avoid repeated preemption of longer-running tasks and can be a useful default strategy. However, submit longer-running jobs directly to an On-Demand queue whenever possible to avoid the unnecessary cost of the initial preemption. ### Consider enabling Fusion Snapshots (preview feature) Fusion Snapshots can reduce interruption risk by checkpointing task state before termination. Fusion Snapshots is in preview and best suited for compute-intensive or long-running tasks. To test this feature, contact the support team at https://support.seqera.io. --- ## Azure Batch :::note This guide assumes you already have an Azure account with a valid Azure Subscription. For details, visit [Azure Free Account][azure-free]. Ensure you have sufficient permissions to create resource groups, an Azure Storage account, and an Azure Batch account. ::: ## Azure concepts #### Regions Azure regions are specific geographic locations around the world where Microsoft has established data centers to host its cloud services. Each Azure region is a collection of data centers that provide users with high availability, fault tolerance, and low latency for cloud services. Each region offers a wide range of Azure services that can be chosen to optimize performance, ensure data residency compliance, and meet regulatory requirements. Azure regions also enable redundancy and disaster recovery options by allowing resources to be replicated across different regions, enhancing the resilience of applications and data. #### Resource groups An Azure resource group is a logical container that holds related Azure resources such as virtual machines, storage accounts, databases, and more. A resource group serves as a management boundary to organize, deploy, monitor, and manage the resources within it as a single entity. Resources in a resource group share the same lifecycle, meaning they can be deployed, updated, and deleted together. This also enables easier access control, monitoring, and cost management, making resource groups a foundational element in organizing and managing cloud infrastructure in Azure. #### Accounts Azure uses accounts for each service. For example, an [Azure Storage account][azure-storage-account] will house a collection of blob containers, file shares, queues, and tables. An Azure subscription can have multiple Azure Storage and Azure Batch accounts - however, a Platform compute environment can only use one of each. Multiple compute environments can be created to use separate credentials, Azure Storage accounts, and Azure Batch accounts. At a minimum, you will require an Azure Batch account and an Azure storage account to run pipelines with Azure Batch with Seqera. This is because Azure uses accounts for each service. #### Service principals An Azure service principal is an identity created specifically for applications, hosted services, or automated tools to access Azure resources. It acts like a user identity with a defined set of permissions, enabling resources authenticated through the service principal to perform actions within the Azure account. Seqera can utilize an Azure service principal to authenticate and access Azure Batch for job execution and Azure Storage for data management. ## Create Azure resources ### Resource group Create a resource group to link your Azure Batch and Azure Storage account: :::note A resource group can be created while creating an Azure Storage account or Azure Batch account. ::: 1. Log in to your Azure account, go to the [Create Resource group][azure-create-rg] page, and select **Create new resource group**. 2. Enter a name for the resource group, such as *seqeracompute*. 3. Choose the preferred region. 4. Select **Review and Create** to proceed. 5. Select **Create**. ### Storage account After creating a resource group, set up an [Azure Storage account][azure-storage-account]: 1. Log in to your Azure account, go to the [Create storage account][azure-create-storage] page, and select **Create a storage account**. :::note If you haven't created a resource group, you can do so now. ::: 2. Enter a name for the storage account, such as *seqeracomputestorage*. 3. Choose the preferred region. This must be the same region as the Batch account. 4. Platform supports all performance or redundancy settings. Select the most appropriate settings for your use case. 5. Select **Next: Advanced**. 6. Enable *storage account key access*. 7. Select **Next: Networking**. - Enable public access from all networks. You can enable public access from selected virtual networks and IP addresses, but you will be unable to use Forge to create compute resources. Disabling public access is not supported. 8. Select **Data protection**. - Configure appropriate settings. All settings are supported by Platform. 9. Select **Encryption**. - Only Microsoft-managed keys (MMK) are supported. 10. In **tags**, add any required tags for the storage account. 11. Select **Review and Create**. 12. Select **Create** to create the Azure Storage account. - You will need at least one Blob Storage container to act as a working directory for Nextflow. 13. Go to your new storage account and select **+ Container** to create a new Blob Storage container. A new container dialog opens. Enter a suitable name, such as *seqeracomputestorage-container*. 14. Go to the **Access Keys** section of your new storage account (*seqeracomputestorage* in this example). 15. Store the access keys for your Azure Storage account, to be used when you create a compute environment. :::caution Blob container storage credentials are associated with the Batch pool configuration. Avoid changing these credentials in Platform after you have created the compute environment. ::: ### Batch account After you have created a resource group and Storage account, create a [Batch account][azure-batch-account]: 1. Log in to your Azure account and select **Create a batch account** on [this page][azure-create-batch]. 2. Select the existing resource group or create a new one. 3. Enter a name for the Batch account, such as *seqeracomputebatch*. 4. Choose the preferred region. This must be the same region as the Storage account. 5. Select **Advanced**. 6. For **Pool allocation mode**, select **Batch service**. 7. For **Authentication mode**, select *Shared Key*. - Microsoft Entra ID is now the recommended credential mechanism for Azure authentication where supported. Use Shared Key here only if your setup requires it for compute environment configuration. 8. Select **Networking**. Ensure networking access is sufficient for Platform and any additional required resources. 9. Add any **Tags** to the Batch account, if needed. 10. Select **Review and Create**. 11. Select **Create**. 12. Go to your new Batch account, then select **Access Keys**. 13. Store the access keys for your Azure Batch account, to be used when you create a Seqera compute environment. :::caution A newly-created Azure Batch account may not be entitled to create virtual machines without making a service request to Azure. See [Azure Batch service quotas and limits][azure-batch-quotas] for more information. ::: 14. Select the **+ Quotas** tab of the Azure Batch account to check and increase existing quotas if necessary. 15. Select **+ Request quota increase** and add the quantity of resources you require. Here is a brief guideline: - **Active jobs and schedules**: Each Nextflow process will require an active Azure Batch job per pipeline while running, so increase this number to a high level. See [here][azure-batch-jobs] to learn more about jobs in Azure Batch. - **Pools**: Each platform compute environment requires at least one Azure Batch pool. Batch Forge creates two pools by default (one for the head job and one for compute tasks). Each pool is composed of multiple machines of one virtual machine size. - **Batch accounts per region per subscription**: Set this to the number of Azure Batch accounts per region per subscription. Only one is required. - **Spot/low-priority vCPUs**: Platform does not support spot or low-priority machines when using Forge, so when using Forge this number can be zero. When manually setting up a pool, select an appropriate number of concurrent vCPUs here. - **Total Dedicated vCPUs per VM series**: See the Azure documentation for [virtual machine sizes][azure-vm-sizes] to help determine the machine size you need. We recommend the latest version of the ED series available in your region as a cost-effective and appropriately-sized machine for running Nextflow. However, you will need to select alternative machine series that have additional requirements, such as those with additional GPUs or faster storage. Increase the quota by the number of required concurrent CPUs. In Azure, machines are charged per cpu minute so there is no additional cost for a higher number. ### Credentials There are two types of Azure credentials available: access keys and Entra service principals. Access keys are simple to use but have several limitations: - Access keys are long-lived. - Access keys provide full access to the Azure Storage and Azure Batch accounts. - Azure allows only two access keys per account, making them a single point of failure. - Access keys do not support VNet/subnet configuration. Entra service principals are accounts which can be granted access to Azure Batch and Azure Storage resources: - Service principals enable role-based access control with more precise permissions. - Service principals map to a many-to-many relationship with Azure Batch and Azure Storage accounts. - Some Azure Batch features, such as VNet/subnet configuration, are only available when using a service principal. Both credential types support Batch Forge and Manual compute environment modes. :::note The two Azure credential types use different authentication methods. You can add more than one credential to a workspace, but Platform compute environments use only one credential at any given time. While separate credentials can be used by separate compute environments concurrently, they are not cross-compatible — access granted by one credential will not be shared with the other. ::: #### Access keys To create an access key: 1. Navigate to the Azure Portal and sign in. 2. Locate the Azure Batch account and select **Keys** under **Account management**. The Primary and Secondary keys are listed here. Copy one of the keys and save it in a secure location for later use. 3. Locate the Azure Storage account and, under the **Security and Networking** section, select **Access keys**. Key1 and Key2 options are listed here. Copy one of them and save it in a secure location for later use. 4. In your Platform workspace **Credentials** tab, select the **Add credentials** button and complete the following fields: - Enter a **Name** for the credentials - **Provider**: Azure - Select the **Shared key** tab - Add the **Batch account** and **Blob Storage account** names and access keys to the relevant fields. 5. Delete the copied keys from their temporary location after they have been added to a credential in Platform. #### Entra service principal and managed identity To use Entra for authentication, you must create a service principal and managed identity. Seqera uses the service principal to authenticate to Azure Batch and Azure Storage. It submits a Nextflow task as the head process to run Nextflow, which authenticates to Azure Batch and Storage using the managed identity attached to the node pool. Therefore, you must create both an Entra service principal and a managed identity: 1. Add the service principal details as credentials in Seqera Platform. 2. Assign the managed identity to each Azure Batch node pool with the relevant permissions. 3. When using Batch Forge, provide the managed identity resource ID for each managed identity. Seqera Platform assigns the identity to each pool during creation. :::note Entra service principal credentials support both Batch Forge and Manual compute environments. Some features, such as VNet/subnet configuration and managed identities, require Entra credentials. When using Entra credentials, a managed identity is recommended for best security practices, but is not mandatory. ::: ##### Service principal See [Create a service principal][azure-create-sp] for more details. To create an Entra service principal: 1. In the Azure Portal, navigate to **Microsoft Entra ID**. Under **App registrations**, select **New registration**. 2. Provide a name for the application. The application will automatically have a service principal associated with it. 3. Assign roles to the service principal: 1. Go to the Azure Storage account. Under **Access Control (IAM)**, select **Add role assignment**. 2. Select the **Storage Blob Data Contributor** role. 3. Select **Members**, then **Select Members**. Search for your newly created service principal and assign the role. 4. Repeat the same process for the Azure Batch account, using the **Azure Batch Data Contributor** role. This role is sufficient for pool creation and is narrower than the general **Azure Batch Account Contributor** role. 5. If you create a managed identity (recommended), also assign the **Managed Identity Operator** role to the service principal on each managed identity. Without this role, Seqera cannot attach the managed identity to a Batch pool. 6. If you plan to deploy Batch pools into a private VNet (by specifying a Subnet ID when creating the compute environment), also assign the Network Contributor role (or a custom role granting `Microsoft.Network/virtualNetworks/subnets/join/action`) to the service principal on the VNet. Only the service principal needs VNet permissions (the head and pool managed identities do not). 4. Platform will need credentials to authenticate as the service principal: 1. Navigate back to the app registration. On the **Overview** page, save the **Application (client) ID** value for use in Platform. 2. Select **Certificates & secrets**, then **New client secret**. A new secret is created containing a value and secret ID. Save both values securely for use in Platform. 5. In your Platform workspace **Credentials** tab, select the **Add credentials** button and complete the following fields: - Enter a **Name** for the credentials - **Provider**: Azure - Select the **Entra** tab - Complete the remaining fields: **Batch account name**, **Blob Storage account name**, **Tenant ID** (Directory (tenant) ID in Azure), **Client ID** (Application (client) ID in Azure), **Client secret** (Client secret value in Azure). 6. Delete the ID and secret values from their temporary location after they have been added to a credential in Platform. ##### Managed identity :::info To use managed identities, Platform requires Nextflow version 24.06.0-edge or later. ::: Nextflow can authenticate to Azure services using a managed identity. This method offers enhanced security compared to access keys, but it must run on Azure infrastructure and requires Entra service principal credentials. Pool creation with a managed identity attached uses the Azure Batch management plane, which only accepts Entra (AAD) tokens, so shared-key credentials cannot create pools with managed identities. When you use a compute environment with a managed identity attached to the Azure Batch pool, Nextflow uses this managed identity for authentication. Seqera still uses the Entra service principal to submit the initial Nextflow task; that task then proceeds with the managed identity for subsequent authentication. When you don't attach a head managed identity, Platform passes the service principal credentials to the head job so it can authenticate to Azure Storage and Azure Container Registry at runtime. This places a long-lived secret on the compute node. Attaching a user-assigned managed identity removes that secret — the VM obtains short-lived tokens from the Azure Instance Metadata Service instead. For this reason, a managed identity is recommended for production deployments. The same applies to the worker pool managed identity used by compute tasks. 1. In Azure, create a user-assigned managed identity. See [Manage user-assigned managed identities][azure-managed-identity] for detailed steps. Take note of both the **client ID** and the **resource ID** of the managed identity when you create it. 2. Assign the following roles to the managed identity: - **Storage Blob Data Contributor** on the Azure Storage account, so the pool VMs can read inputs and write outputs. - **AcrPull** on any Azure Container Registry the pipeline pulls images from. Without this role, container pulls fail when the pool VM authenticates via the managed identity. See [Required role assignments][nf-azure-roles] for more information. 3. Associate the user-assigned managed identity with the Azure Batch pool. See [Set up managed identity in your Batch pool][azure-batch-mi-pool] for more information. :::note When you use separate head and worker pools, you can assign a different managed identity to each pool. Typically, the head managed identity needs broader Batch and storage permissions, while the worker managed identity only needs storage and `AcrPull` access. ::: 4. When you set up the Seqera compute environment, provide the managed identity details in the specified fields. The form has four managed identity fields — a **client ID** and a **resource ID** for both the head pool and the worker pool: - **Resource IDs** are the full ARM paths of the managed identities (e.g., `/subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identityName}`). Seqera passes these to Azure Batch at pool-create time to attach the managed identity to the head pool and worker pool VMs respectively. Resource IDs are required when using Batch Forge. - **Client IDs** are passed to Nextflow, Fusion, and AzCopy on the pool VMs. The Azure Instance Metadata Service uses the client ID to mint a token for the correct managed identity. A VM can have multiple managed identities attached, so the consumer must specify which one to use. You can use the same managed identity for both head and worker pools by entering the same values in both pairs of fields, but separate identities are recommended so that worker RBAC can stay narrower than head RBAC. The four fields work for both single-pool and dual-pool topologies: - **Single-pool** (head pool only): both managed identities are attached to the same VMs. The client IDs are required to disambiguate which managed identity each consumer authenticates as — when a VM has more than one user-assigned managed identity, the Azure Instance Metadata Service (IMDS) does not pick a default, so each consumer must pass its UAMI's `client_id` on the IMDS request. - **Dual-pool** (separate head and worker pools): each pool has only its own managed identity attached, so disambiguation is implicit at the VM level. The client IDs are still required so that consumers know which managed identity is theirs. When you submit a pipeline to this compute environment, Nextflow authenticates using the managed identity associated with the Azure Batch node it runs on, rather than relying on access keys. :::caution If a managed identity is misconfigured (for example, an invalid client ID or missing RBAC roles), the pipeline fails with an explicit error. Seqera does not silently fall back to the service principal at runtime. ::: ## Add compute environment There are two ways to create an Azure Batch compute environment in Platform: - [**Batch Forge**](#batch-forge): Automatically creates Azure Batch resources. - [**Manual**](#manual): For using existing Azure Batch resources. :::tip For an end-to-end walkthrough of Azure Batch topologies (separate head and worker pools, Entra authentication, and private networking), see [Set up advanced Azure Batch compute environments](../enterprise/advanced-topics/manual-azure-batch-setup). ::: ### VM size considerations Azure Batch requires you to select an appropriate VM size for your compute environment. There are a number of considerations when selecting VM sizes. See [Sizes for virtual machines in Azure][azure-vm-sizes-overview] for more information. 1. **Family**: The first letter of the VM size name indicates the machine family. For example, `Standard_E16d_v5` is a member of the E family. - *A*: Economical machines, low power machines. - *B*: Burstable machines which use credits for cost allocation. - *D*: General purpose machines suitable for most applications. - *DC*: D machines with additional confidential compute capabilities. - *E*: The same as D but with more memory. These are generally the best machines for bioinformatics workloads. - *EC*: The same as E but with additional confidential compute capabilities. - *F*: Compute optimized machines which come with a faster CPU compared to D-series machines. - *M*: Memory optimized machines which come with extremely large and fast memory layers, typically more than is needed for bioinformatics workloads. - *L*: Storage optimized machines which come with large locally attached NVMe storage drives. Note that these need to be configured before you can use them with Azure Batch. - *N*: Accelerated computing machines which come with FPGAs, GPUs, or custom ASICs. - *H*: High performance machines which come with the fastest processors and memory. In general, we recommend using the E family of machines for bioinformatics workloads since these are cost-effective, widely available, and sufficiently fast. 1. **vCPUs**: The machine's number of vCPUs. This is the main factor in determining the speed of the machine. 2. **features**: Additional machine features. For example, some machines come with a local SSD. - d: A local temp disk. Azure Batch uses this disk for task working directories, and Batch Forge pools can also use it for container data. - s: The VM supports a [premium storage account][azure-premium-storage]. - a: AMD CPUs instead of Intel. - p: ARM-based CPUs, such as Azure Cobalt. - l: Reduced memory with a large cost reduction. 3. **Version**: The version of the VM size. This is the generation of the machine. Typically, more recent is better but availability can vary between regions. In the Azure Portal on the page for your Azure Batch account, request an appropriate quota for your desired VM size. See [Azure Batch service quotas and limits][azure-batch-quotas] for more information. :::note When a VM size has a local temp disk (`/mnt`) of at least 128 GB that is at least as large as the boot disk, Batch Forge pools store Docker and containerd data on the temp disk instead of the OS disk. The temp disk is faster than the default OS disk, which keeps container creation within the Batch node agent's 5-minute limit on busy nodes. Many `d` sizes from v4 and v5 qualify, such as `Standard_E16d_v5`. NVMe-only v6 sizes don't. Container images share the temp disk with task working directories. Allow for both when you choose a VM size. Existing compute environments keep container data on the OS disk. To use the temp disk, create the compute environment again. ::: ### Batch Forge :::caution Batch Forge automatically creates resources that you may be charged for in your Azure account. See [Cloud costs][cloud-costs] for guidelines to manage cloud resources effectively and prevent unexpected costs. ::: Create a Batch Forge Azure Batch compute environment: 1. In a workspace, select **Compute Environments > New Environment**. 2. Enter a descriptive name, such as *Azure Batch (east-us)*. 3. Select **Azure Batch** as the target platform. 4. Choose existing Azure credentials or add a new credential. :::note Both access keys and Entra service principal credentials are supported for Batch Forge. Some features, such as VNet/subnet configuration, require Entra credentials. ::: 5. Add the **Batch account** and **Blob Storage** account names and access keys. 6. Select a **Region**, such as *eastus*. 7. In the **Work directory** field, enter the Azure blob container created previously. For example, `az://seqeracomputestorage-container/work`. :::note When you specify a Blob Storage bucket as your work directory, this bucket is used for the Nextflow [cloud cache][nf-cloud-cache] by default. You can specify an alternative cache location with the **Nextflow config file** field on the pipeline [launch][launch-form] form. ::: 8. Select **Enable Wave containers** to facilitate access to private container repositories and provision containers in your pipelines using the Wave containers service. See [Wave containers][wave] for more information. 9. Select **Enable Fusion v2** to allow access to your Azure Blob Storage data via the [Fusion v2][fusion] virtual distributed file system. This speeds up most data operations. The Fusion v2 file system requires Wave containers to be enabled. See [Fusion file system][fusion-overview] for configuration details.
Use Fusion v2 :::note The compute recommendations below are based on internal benchmarking performed by Seqera. Benchmark runs of [nf-core/rnaseq](https://github.com/nf-core/rnaseq) used profile `test_full`, consisting of an input dataset with 16 FASTQ files and a total size of approximately 123.5 GB. ::: Azure virtual machines include fast SSDs and require no additional storage configuration for Fusion. For optimal performance, use VMs with sufficient local storage to support Fusion's streaming data throughput. 1. Use Seqera Platform version 23.1 or later. 2. Use an Azure Blob storage container as the work directory. 3. Enable **Wave containers** and **Fusion v2**. 4. Select the **Batch Forge** config mode. 5. Specify suitable VM sizes under **VMs type**. A `Standard_E16d_v5` VM or larger is recommended for production use. :::tip We recommend selecting machine types with a local temp storage disk of at least 200 GB and a random read speed of 1000 MBps or more for large and long-lived production pipelines. To work with files larger than 100 GB, increase temp storage accordingly (400 GB or more). On Batch Forge pools, container images also use this disk. The suffix `d` after the core number (e.g., `Standard_E16*d*_v5`) denotes a VM with a local temp disk. Select instances with Standard SSDs — Fusion does not support Azure network-attached storage (Premium SSDv2, Ultra Disk, etc.). Larger local storage increases Fusion's throughput and reduces the chance of overloading the machine. See [Sizes for virtual machines in Azure][azure-vm-sizes-overview] for more information. :::
10. (Optional) Enter a **Subnet ID** to connect the Batch pool nodes to a private Azure VNet. Enter the full Azure ARM subnet resource ID in the following format: ``` /subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.Network/virtualNetworks/{vnetName}/subnets/{subnetName} ``` :::note VNet/subnet configuration requires Entra credentials. This field is only available when you select Entra credentials. If you do not provide a subnet ID, default networking is used. The service principal must have the **Network Contributor** role (or `Microsoft.Network/virtualNetworks/subnets/join/action`) on the VNet. Otherwise, pool creation fails. ::: 11. Set the **Config mode** to **Batch Forge**. 12. Enter the default **VMs type** for compute tasks, depending on your quota limits set previously. The default is _Standard_D4_v3_. :::note See [VM size considerations](#vm-size-considerations) for how Batch Forge uses the temp disk of the VM size you select. ::: 13. Enter the **VMs count**. If autoscaling is enabled (default), this is the maximum number of VMs the compute pool will scale up to. If autoscaling is disabled, this is the fixed number of virtual machines in the compute pool. 14. (Optional) Configure **Head job resources** to control the VM type and resources allocated to the Nextflow head job: - **Head VM type**: The VM size for the head node pool. If not specified, the same VM type as the compute pool is used. - **Head job CPUs**: The number of CPUs allocated to the Nextflow head job. - **Head job memory**: The amount of memory allocated to the Nextflow head job. 15. Enable **Autoscale** to scale the compute pool up and down automatically, based on the number of pipeline tasks. The number of VMs will vary from **0** to **VMs count**. 16. Enable **Dispose resources** for Seqera to automatically delete the Batch pools if the compute environment is deleted on the platform. :::info Batch Forge creates separate Azure Batch pools for the Nextflow head job and compute tasks by default (named `tower-pool-{envId}-head` and `tower-pool-{envId}-worker`). This prevents the head node from competing for resources with compute tasks and allows independent sizing of each pool. See [Batch service quotas and limits](https://learn.microsoft.com/en-us/azure/batch/batch-quota-limit) for information about limits when running multiple compute environments. ::: 17. Select or create [**Container registry credentials**][azure-registry-credentials] to authenticate a registry (used by the [Wave containers][nf-wave] service). It is recommended to use an [Azure Container registry][azure-container-registry] within the same region for maximum performance. 18. Apply [**Resource labels**][resource-labels]. This will populate the **Metadata** fields of the Azure Batch pools and jobs. 19. Expand **Staging options** to include: - Optional [pre- or post-run Bash scripts][pre-post-run-scripts] that execute before or after the Nextflow pipeline execution in your environment. - Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the **Nextflow config file** field in the pipeline launch form. These values can be overridden during pipeline launch. :::info Configuration settings in this field override the same values in the pipeline repository `nextflow.config` file. See [Nextflow config file][nf-config-file] for more information on configuration priority. ::: 20. Under **Environment variables**, add each variable with a **Name**, **Value**, and **Target Environment**: - **Head job**: Adds the variable to the Nextflow head job container, which evaluates `nextflow.config` and submits tasks to the compute backend. Use this target for variables that Nextflow or its plugins read, such as `NXF_OPTS`, `NXF_JVM_ARGS`, `NXF_PLUGINS_DEFAULT`, or proxy settings the head node uses to reach external services. - **Compute job**: Adds the variable to the worker containers that run individual pipeline tasks. Use this target for variables your pipeline tools read, such as `OPENAI_API_KEY` for a process that calls the OpenAI API, registry credentials needed inside the task container, or tool-specific settings like `JAVA_HOME`. - **Head and Compute jobs**: Adds the variable to both the head job and the compute jobs. Use this target for values needed in both places, such as an HTTP proxy used by both Nextflow and task tools, or a credential needed in both the head job and individual compute tasks. :::note For sensitive values such as API keys and tokens, use [pipeline secrets](../secrets/overview) instead of custom environment variables. Custom environment variables are stored in the compute environment configuration and cannot be edited after creation. To rotate a value, recreate the compute environment. ::: 21. Configure any advanced options you need: - Use **Max wallclock time** to set the maximum duration a job can run. The default is 7 days. Accepts human-readable duration syntax (e.g., `7d`, `12h`, `1d6h30m`). The maximum allowed by Azure Batch is 180 days. Existing compute environments without this setting use Nextflow's default of 30 days. - **Job cleanup toggles** control how Nextflow process jobs are managed on completion. Active jobs consume the quota of your Azure Batch account. Three independent toggles are available: | Toggle | Default | Description | |--------|---------|-------------| | **Delete jobs on completion** | Off | Permanently deletes all jobs and their tasks from Azure Batch when the workflow finishes. | | **Delete tasks on completion** | On | Deletes individual tasks from jobs when they complete successfully. Failed tasks are preserved for debugging. | | **Terminate jobs on completion** | On | Sets jobs to terminate when all their tasks complete. Jobs remain in "completed" state but are no longer active. | Existing compute environments retain their current cleanup behavior. - Use **Token duration** to control the duration of the SAS token generated by Nextflow. This must be as long as the longest period of time the pipeline will run. 22. Select **Add** to finalize the compute environment setup. It will take a few seconds for all the resources to be created before the compute environment is ready to launch pipelines. :::info See [Launch pipelines][launch-pipelines] to start executing workflows in your Azure Batch compute environment. ::: ### Manual You can configure Seqera Platform to use a pre-existing Azure Batch pool. This allows the use of more advanced Azure Batch features, such as custom VM images and private networking. See [Azure Batch security best practices][azure-batch-security] for more information. :::caution Your Seqera compute environment uses resources that you may be charged for in your Azure account. See [Cloud costs][cloud-costs] for guidelines to manage cloud resources effectively and prevent unexpected costs. ::: #### Create a Nextflow-compatible Azure Batch pool If not described below, use the default settings: 1. **Account**: You must have an existing Azure Batch account. Ideally, you should already have tested whether you can run an Azure Batch task within this account. Any type of account is compatible. 2. **Quota**: You must check you have sufficient quota for the number of pools, jobs, and vCPUs per series. See [Azure Batch service quotas and limits][azure-batch-quotas] for more information. 3. On the Azure Batch page of the Azure Portal, select **Pools** and then **+ Add**. 4. **Name**: Enter a **Pool ID** and **Display Name**. This ID will be used by Seqera and Nextflow. 5. **Identity**: Select **User assigned** to use a managed identity for the pool. Select **Add** for the user-assigned managed identity and select the managed identity with the correct permissions to the Azure Storage and Batch accounts. 6. **Operating System**: You can use any Linux-based image here, but it is recommended to use it with a Microsoft Azure Batch-provided image. Note that there are two generations of Azure Virtual Machine images, and certain VM series are only available in one generation. See [Azure Virtual Machine series][azure-vm-gen2] for more information. For default settings, select the following: - **Publisher**: `microsoft-dsvm` - **Offer**: `ubuntu-hpc` - **Sku**: `2204` - **Security type**: `standard` 7. **OS disk storage account type**: Certain VM series only support a specific Storage account type. See [Azure managed disk types][azure-disk-types] and [Azure Virtual Machine series][azure-vm-gen2] for more information. In general, a VM series with the suffix *s* supports a *Premium LRS* Storage account type. For example, a `standard_e16ds_v5` supports `Premium_LRS` but a `standard_e16d_v5` does not. Premium LRS offers the best performance. 8. **OS disk size**: The size of the OS disk in GB. This must be sufficient to hold every Docker container the VM will run, plus any logging or further files. If you are not using a machine with attached storage, you must increase this disk size to accommodate task files (see VM type below). If you are using a machine with attached storage, this setting can be left at the OS default size. 9. **Container configuration**: Container configuration must be turned on. Do this by switching it from **None** to **Custom**. The type is **Docker compatible** which should be the only available option. This will enable the VM to use Docker images and is sufficient. However, you can add further options: - Under **Container image names** you can add containers for the VM to grab at startup time. Add a list of fully qualified Docker URIs, such as `quay.io/seqeralabs/nf-launcher:j17-23.04.2`. - Under **Container registries**, you can add any container registries that require additional authentication. Select **Container registries**, then **Add**. Here, you can add a registry username, password, and registry server. If you attached the managed identity earlier, select this as an authentication method so you don't have to enter a username and password. 10. **VM size**: This is the size of the VM. See [Sizes for virtual machines in Azure][azure-vm-sizes] for more information. 11. **Scale**: Azure Node pools can be fixed in size or autoscale based on a formula. Autoscaling is recommended to enable scaling your resources down to zero when not in use. Select **Auto scale** and change the **AutoScale evaluation interval** to 5 minutes - this is the minimum period between evaluations of the autoscale formula. For **Formula**, you can use any valid formula — See [Create a formula to automatically scale compute nodes in a Batch pool][azure-autoscale] for more information. This is the default autoscaling formula, with a maximum of 8 VMs: ``` // Get pool lifetime since creation. lifespan = time() - time("2024-10-30T00:00:00.880011Z"); interval = TimeInterval_Minute * 5; // Compute the target nodes based on pending tasks. // $PendingTasks == The sum of $ActiveTasks and $RunningTasks $samples = $PendingTasks.GetSamplePercent(interval); $tasks = $samples < 70 ? max(0, $PendingTasks.GetSample(1)) : max( $PendingTasks.GetSample(1), avg($PendingTasks.GetSample(interval))); $targetVMs = $tasks > 0 ? $tasks : max(0, $TargetDedicatedNodes/2); targetPoolSize = max(0, min($targetVMs, 8)); // For first interval, deploy 1 node, for other intervals scale up/down as per tasks. $TargetDedicatedNodes = lifespan < interval ? 1 : targetPoolSize; $NodeDeallocationOption = taskcompletion; ``` 12. **Start task**: This is the task that will run on each VM when it joins the pool. This can be used to install additional software on the VM. When using Batch Forge, this is used to install `azcopy` for staging files onto and off of the node. Select **Enabled** and add the following command line to install `azcopy`: ```shell bash -c "chmod +x azcopy && mkdir $AZ_BATCH_NODE_SHARED_DIR/bin/ && cp azcopy $AZ_BATCH_NODE_SHARED_DIR/bin/" ``` Select **Resource files** then select **Http url**. For the **URL**, add `https://nf-xpack.seqera.io/azcopy/linux_amd64_10.8.0/azcopy` and for **File path** enter `azcopy`. Every other setting can be left default. :::note When not using Fusion, every node **must** have `azcopy` installed. ::: 13. **Task Slots**: Set task slots to the machine's number of vCPUs. For example, select `4` for a `Standard_D4_v3` VM size. 14. **Task scheduling policy**: This can be set to `Pack` or `Spread`. `Pack` will attempt to schedule tasks from the same job on the same VM, while `Spread` will attempt to distribute tasks evenly across VMs. 15. **Virtual Network**: If you are using a virtual network, you can select it here. Be sure to select the correct virtual network and subnet. The VMs require: - Access to container registries (such as quay.io and docker.io) to pull containers. - Access to Azure Storage to copy data using `azcopy`. - Access to any remote files required by the pipeline, such as AWS S3 storage. - Communication with the head node that runs Nextflow and Seqera to relay logs and information. Note that overly-restrictive networking may prevent pipelines from running successfully. 16. **Mount configuration**: Nextflow *only* supports Azure File Shares. Select `Azure Files Share`, then add: - **Source**: URL in format `https://${accountName}.file.core.windows.net/${fileShareName}` - **Relative mount path**: Path where the file share will be mounted on the VM - **Storage account name** and **Storage account key** (managed identity is not supported) Leave the node pool to start and create a single Azure VM. Monitor the VM to ensure it starts correctly. If any errors occur, check and correct them - you may need to create a new Azure node pool if issues persist. The following settings can be modified after creating a pool: - Autoscale formula - Start task - Application packages - Node communication - Metadata #### Create a manual Seqera Azure Batch compute environment 1. In a workspace, select **Compute Environments**, then **Add compute environment**. 2. Enter a descriptive name for this environment, such as *Azure Batch (east-us)*. 3. For **Provider**, select **Azure Batch**. 4. Select your existing Azure credentials (access keys or Entra service principal) or select **+** to add new credentials. :::note Both access keys and Entra service principal credentials are supported. Some features, such as VNet/subnet configuration, require Entra credentials. To use Entra with a managed identity, see [Managed identity](#managed-identity) below. ::: 5. Select a **Region**, such as *eastus (East US)*. 6. In the **Work directory** field, add the Azure blob container created previously. For example, `az://seqeracomputestorage-container/work`. :::note When you specify a Blob Storage bucket as your work directory, this bucket is used for the Nextflow [cloud cache][nf-cloud-cache] by default. You can specify an alternative cache location with the **Nextflow config file** field on the pipeline [launch][launch-form] form. ::: 7. Select **Enable Wave containers** to facilitate access to private container repositories and provision containers in your pipelines using the Wave containers service. See [Wave containers][wave] for more information. 8. Select **Enable Fusion v2** to allow access to your Azure Blob Storage data via the [Fusion v2][fusion] virtual distributed file system. This speeds up most data operations. The Fusion v2 file system requires Wave containers to be enabled. See [Fusion file system][fusion-overview] for configuration details.
Use Fusion v2 :::note The compute recommendations below are based on internal benchmarking performed by Seqera. Benchmark runs of [nf-core/rnaseq](https://github.com/nf-core/rnaseq) used profile `test_full`, consisting of an input dataset with 16 FASTQ files and a total size of approximately 123.5 GB. ::: Azure virtual machines include fast SSDs and require no additional storage configuration for Fusion. For optimal performance, use VMs with sufficient local storage to support Fusion's streaming data throughput. 1. Use Seqera Platform version 23.1 or later. 2. Use an Azure Blob storage container as the work directory. 3. Enable **Wave containers** and **Fusion v2**. 4. Specify suitable VM sizes under **VMs type**. A `Standard_E16d_v5` VM or larger is recommended for production use. :::tip We recommend selecting machine types with a local temp storage disk of at least 200 GB and a random read speed of 1000 MBps or more for large and long-lived production pipelines. To work with files larger than 100 GB, increase temp storage accordingly (400 GB or more). The suffix `d` after the core number (e.g., `Standard_E16*d*_v5`) denotes a VM with a local temp disk. Select instances with Standard SSDs — Fusion does not support Azure network-attached storage (Premium SSDv2, Ultra Disk, etc.). Larger local storage increases Fusion's throughput and reduces the chance of overloading the machine. See [Sizes for virtual machines in Azure][azure-vm-sizes-overview] for more information. :::
9. (Optional) Enter a **Subnet ID** to connect the Batch pool nodes to a private Azure VNet. Enter the full Azure ARM subnet resource ID in the following format: ``` /subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.Network/virtualNetworks/{vnetName}/subnets/{subnetName} ``` :::note VNet/subnet configuration requires Entra credentials. This field is only available when you select Entra credentials. If you do not provide a subnet ID, default networking is used. The service principal must have the **Network Contributor** role (or `Microsoft.Network/virtualNetworks/subnets/join/action`) on the VNet. Otherwise, pool creation fails. ::: 10. Set the **Config mode** to **Manual**. 11. Enter the **Head Pool name**. This is the name of the Azure Batch pool for the Nextflow head job. 12. (Optional) Enter a **Compute Pool name** for worker tasks. If not specified, the head pool is used for both head and compute tasks (single-pool mode). :::tip Using separate pools for head and compute nodes allows you to right-size each pool independently. For example, you can use a smaller VM for the Nextflow head job and larger VMs for compute-intensive tasks. ::: 13. Enter user-assigned **Managed identity client IDs** (and optionally **resource IDs**), if managed identities are attached to your Azure Batch pools. In dual-pool mode, you can specify separate managed identities for the head and compute pools. See [Managed Identity](#managed-identity) below. 14. Apply [**Resource labels**][resource-labels]. This will populate the **Metadata** fields of the Azure Batch pools and jobs. 15. Expand **Staging options** to include: - Optional [pre- or post-run Bash scripts][pre-post-run-scripts] that execute before or after the Nextflow pipeline execution in your environment. - Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the **Nextflow config file** field in the pipeline launch form. These values can be overridden during pipeline launch. :::info Configuration settings in this field override the same values in the pipeline repository `nextflow.config` file. See [Nextflow config file][nf-config-file] for more information on configuration priority. ::: 16. Under **Environment variables**, add each variable with a **Name**, **Value**, and **Target Environment**: - **Head job**: Adds the variable to the Nextflow head job container, which evaluates `nextflow.config` and submits tasks to the compute backend. Use this target for variables that Nextflow or its plugins read, such as `NXF_OPTS`, `NXF_JVM_ARGS`, `NXF_PLUGINS_DEFAULT`, or proxy settings the head node uses to reach external services. - **Compute job**: Adds the variable to the worker containers that run individual pipeline tasks. Use this target for variables your pipeline tools read, such as `OPENAI_API_KEY` for a process that calls the OpenAI API, registry credentials needed inside the task container, or tool-specific settings like `JAVA_HOME`. - **Head and Compute jobs**: Adds the variable to both the head job and the compute jobs. Use this target for values needed in both places, such as an HTTP proxy used by both Nextflow and task tools, or a credential needed in both the head job and individual compute tasks. :::note For sensitive values such as API keys and tokens, use [pipeline secrets](../secrets/overview) instead of custom environment variables. Custom environment variables are stored in the compute environment configuration and cannot be edited after creation. To rotate a value, recreate the compute environment. ::: 17. Configure any advanced options you need: - Use **Max wallclock time** to set the maximum duration a job can run. The default is 7 days. Accepts human-readable duration syntax (e.g., `7d`, `12h`, `1d6h30m`). The maximum allowed by Azure Batch is 180 days. Existing compute environments without this setting use Nextflow's default of 30 days. - **Job cleanup toggles** control how Nextflow process jobs are managed on completion. Active jobs consume the quota of your Azure Batch account. Three independent toggles are available: | Toggle | Default | Description | |--------|---------|-------------| | **Delete jobs on completion** | Off | Permanently deletes all jobs and their tasks from Azure Batch when the workflow finishes. | | **Delete tasks on completion** | On | Deletes individual tasks from jobs when they complete successfully. Failed tasks are preserved for debugging. | | **Terminate jobs on completion** | On | Sets jobs to terminate when all their tasks complete. Jobs remain in "completed" state but are no longer active. | Existing compute environments retain their current cleanup behavior. - Use **Token duration** to control the duration of the SAS token generated by Nextflow. This must be as long as the longest period of time the pipeline will run. 18. Select **Add** to complete the compute environment setup. The creation of resources will take a few seconds, after which you can launch pipelines. :::info See [Launch pipelines][launch-pipelines] to start executing workflows in your Azure Batch compute environment. ::: ## Fusion on Azure Batch with Ubuntu 24.04 The `ubuntu-hpc` image in Azure Batch now defaults to Ubuntu 24.04, whose kernel and AppArmor policy block the unprivileged user namespaces and FUSE mounts Fusion needs. Running Fusion on Ubuntu 24.04+ pools therefore requires an AppArmor profile that permits Fusion mounts to be loaded on every node, plus a matching `--security-opt` on each task container. **Forge compute environments** Seqera Platform handles everything automatically. No action required. **Manual compute environments** When Fusion is enabled and the pool's image SKU is 2404 or higher, Seqera Platform will automatically append the options below to the generated Nextflow configuration: `process.containerOptions = '--security-opt apparmor=seqera-fusionfs-container'` ### Install the AppArmor profile on pool nodes A Fusion-specific seqera-fusionfs-container AppArmor profile must be loaded into the kernel on every node before any Fusion tasks run. The contents of the profile are as follows: ``` abi , include profile seqera-fusionfs-container flags=(default_allow) { userns, mount fstype=fuse.fusion -> /fusion/, mount fstype=fuse.fusion -> /fusion/**, umount, include include include if exists } ``` The profile can be delivered into the nodes by any suitable mechanism: for example, it can be embedded in a custom VM image, distributed via configuration management, or written during node bootstrap. Note that the profile name (`seqera-fusionfs-container`) is mandatory because Seqera Platform code will refer to it, but its location and load mechanism are flexible. Here are some common ways to get the profile loaded: - Drop the file at `/etc/apparmor.d/seqera-fusionfs-container` in a custom VM image. Ubuntu's apparmor.service will automatically load profiles from `/etc/apparmor.d/` at boot, so no explicit loading is needed. - Place the profile anywhere and call `apparmor_parser -r ` to load the profile during node bootstrap (pool start task, cloud-init, configuration management run). This is what Seqera Platform does for Forge-provisioned compute environments. - Place the profile in a subdirectory and configure a boot-time `systemd` unit that runs `apparmor_parser -r` against it on every boot. This is useful if the profile lives in a non-standard location to ensure the profile is re-loaded after every reboot. ### Caveat: overriding `process.containerOptions` A `process.containerOptions` value explicitly set in a compute environment's Nextflow config field will override the one Platform injects. For these cases, including `seqera-fusionfs-container` as a Docker `security-opt` is mandatory: `process.containerOptions = '--security-opt apparmor=seqera-fusionfs-container '` ### Troubleshooting Run the following checks by SSHing into a pool node: ``` # 1.Is the profile loaded into the kernel on this node? aa-status | grep seqera-fusionfs-container # 2. Is the profile applied to a running task container? docker inspect --format '{{.AppArmorProfile}}' ``` | Symptom | Likely cause | |--------|---------| | Workflow task fails with `Operation not permitted` on `/fusion/`, host `dmesg` / `/var/log/kern.log` shows `apparmor="DENIED" ... profile="docker-default"` | The AppArmor profile is loaded on the node, but `--security-opt apparmor=seqera-fusionfs-container` is not being applied. Check whether `process.containerOptions` is overridden in the compute environment's Nextflow config | | `apparmor="DENIED" ... profile="unconfined"` or `aa-status` does not list `seqera-fusionfs-container` | The profile is not loaded on this node. Re-run `apparmor_parser -r ` or fix the boot-time mechanism that should load it. | | `apparmor_parser` itself fails with a syntax error | The profile contents were corrupted in transit (check for stray indentation or encoding changes). Compare against the canonical version above. | | Task container shows `AppArmorProfile: seqera-fusionfs-container` but Fusion still fails to mount | AppArmor is not the issue. Check the Fusion logs and that the FUSE device (`/dev/fuse`) is available inside the container. | [azure-free]: https://azure.microsoft.com/en-us/free/ [azure-storage-account]: https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview [azure-create-rg]: https://portal.azure.com/#create/Microsoft.ResourceGroup [azure-create-storage]: https://portal.azure.com/#create/Microsoft.StorageAccount-ARM [azure-batch-account]: https://learn.microsoft.com/en-us/training/modules/create-batch-account-using-azure-portal/ [azure-create-batch]: https://portal.azure.com/#create/Microsoft.BatchAccount [azure-batch-quotas]: https://docs.microsoft.com/en-us/azure/batch/batch-quota-limit#view-batch-quotas [azure-batch-jobs]: https://learn.microsoft.com/en-us/azure/batch/jobs-and-tasks [azure-vm-sizes]: https://learn.microsoft.com/en-us/azure/virtual-machines/sizes [azure-vm-sizes-overview]: https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/overview [azure-premium-storage]: https://learn.microsoft.com/en-us/azure/virtual-machines/premium-storage-performance [azure-create-sp]: https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal [azure-managed-identity]: https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities [azure-batch-mi-pool]: https://learn.microsoft.com/en-us/troubleshoot/azure/hpc/batch/use-managed-identities-azure-batch-account-pool#set-up-managed-identity-in-your-batch-pool [azure-batch-security]: https://learn.microsoft.com/en-us/azure/batch/security-best-practices [azure-vm-gen2]: https://learn.microsoft.com/en-us/azure/virtual-machines/generation-2 [azure-disk-types]: https://learn.microsoft.com/en-us/azure/virtual-machines/disks-types [azure-autoscale]: https://learn.microsoft.com/en-us/azure/batch/batch-automatic-scaling [azure-container-registry]: https://azure.microsoft.com/en-gb/products/container-registry [nf-azure-roles]: https://docs.seqera.io/nextflow/azure#required-role-assignments [nf-cloud-cache]: https://docs.seqera.io/nextflow/cache-and-resume#cache-stores [nf-wave]: https://docs.seqera.io/nextflow/wave [nf-config-file]: https://docs.seqera.io/platform-enterprise/launch/advanced#nextflow-config-file [wave]: https://docs.seqera.io/wave [fusion]: https://docs.seqera.io/fusion [fusion-overview]: https://docs.seqera.io/platform-enterprise/supported_software/fusion/overview [cloud-costs]: https://docs.seqera.io/platform-enterprise/monitoring/cloud-costs [launch-pipelines]: https://docs.seqera.io/platform-enterprise/launch/launchpad [launch-form]: https://docs.seqera.io/platform-enterprise/launch/launchpad#launch-form [pre-post-run-scripts]: https://docs.seqera.io/platform-enterprise/launch/advanced#pre-and-post-run-scripts [resource-labels]: https://docs.seqera.io/platform-enterprise/resource-labels/overview [azure-registry-credentials]: https://docs.seqera.io/platform-enterprise/credentials/azure_registry_credentials --- ## Azure Cloud :::note This compute environment type is currently in public preview. Please consult this guide for the latest information on recommended configuration and limitations. This guide assumes you already have an Azure account with a valid Azure subscription. ::: Many of the current implementations of compute environments for cloud providers rely on the use of batch services such as AWS Batch, Azure Batch, and Google Batch for the execution and management of submitted jobs, including pipelines and Studio session environments. Batch services are suitable for large-scale workloads, but they add management complexity. In practical terms, the currently used batch services result in some limitations: - **Complex setup**: Azure Batch compute environments require users to independently configure their own Batch accounts. - **Long-lived credentials**: Azure Batch uses access keys to authenticate with Batch and Storage accounts. These credentials are long-lived, and Azure has hard limits on the number of access keys that can be created per resource type. - **Quotas**: Azure Batch accounts have limits for jobs, pools, and compute resources. If these limits are exceeded, no additional pipelines can run until the existing resources are removed. The Azure Cloud compute environment addresses these pain points with: - **Simplified configuration**: Fewer configurable options, with opinionated defaults, provide the best Nextflow pipeline and Studio session execution environment, with both [Wave](https://docs.seqera.io/wave) and [Fusion](https://docs.seqera.io/fusion) enabled. - **More secure credentials**: Authenticate exclusively via [Entra ID](https://learn.microsoft.com/en-us/entra/fundamentals/what-is-entra). This provides enhanced security by default, with automatic configuration for the user. This type of compute environment is best suited to run Studios and small to medium-sized pipelines. It offers more predictable compute pricing, given the fixed instance types. It spins up a standalone virtual machine and executes a Nextflow pipeline or Studio session with a local executor on the virtual machine. At the end of the execution, the instance is terminated. ## Limitations - The Nextflow pipeline will run entirely on a single virtual machine. If the instance does not have sufficient resources, the pipeline execution will fail. For this reason, the number of tasks Nextflow can execute in parallel is limited by the number of cores of the instance type selected. If you need more computing resources, you must create a new compute environment with a larger instance type. This makes the compute environment less suited for larger, more complex pipelines. - There is a considerable delay before streaming logs can be queried. This means that if your pipeline completes in under a minute, you might not see streaming logs during the execution. ## Created resources Seqera will create the following resources in Azure when creating the compute environment: - One Azure resource group: The container for all other created resources. - One Azure managed identity: The Entra identity connected to the Virtual Machine, enabling Nextflow to authenticate to Azure services. - One Azure role: The role attached to the managed identity, which grants the necessary permissions. - One log analytics workspace: Used to collect and query execution logs. - One data collection rule: To route execution logs to the appropriate Log Analytics table. - One data collection endpoint: The endpoint that receives logs, tied to the data collection rule. - One virtual network: The network in which virtual machines are launched. This resource is only created when no existing virtual network is specified in **Advanced options**. When you provide your own VNet, Platform uses it directly and no network resources are provisioned. When virtual machines are launched, other resources are provisioned for each machine and tied to the machine lifecycle: - One network interface - One OS disk While the workflow is running, logs are streamed to the `Nextflow_log_CL` table in the Log Analytics workspace for the compute environment. You can query logs for your specific workflow ID with this expression: ``` Nextflow_log_CL | where workflowId == "" ``` The table retains logs for 7 days. Nextflow uploads log files to Azure Storage for long-term storage. ## Networking Azure Cloud compute environments use a private-only networking model: - **No public IP**: VMs are launched without a public IP address. All connectivity between Platform and the VM is routed via private networking. If you specify an existing VNet, ensure it has outbound connectivity to Azure services (Storage, Entra ID, Log Analytics) and to Platform. - **Studios sessions are outbound-only**: The Seqera Connect client inside a Studio session opens a tunnel outward to the Connect server and registers the session over it. All session traffic, including SSH when enabled, travels over that outbound connection. **No inbound path to the session VM is required.** Users reach a Studio through the Connect proxy rather than by connecting to the VM, so you do not need inbound rules or source-IP allow-lists for dynamically launched Studio VMs. If you specify an existing VNet, ensure it also has outbound connectivity to the Connect server. - **Entra ID only**: Azure Cloud credentials require Microsoft Entra ID (client ID and client secret). Storage account key–based credentials are not supported. This applies to both Forge-provisioned and existing virtual networks. For the ports and directions to configure on your firewall, see [Firewall configuration](../enterprise/advanced-topics/firewall-configuration). {/* TODO: EDU-420 owns the Studios port/direction table on the firewall page. Link the specific anchor once it publishes; do not duplicate the table here. */} ## Requirements ### Platform credentials To create and launch pipelines or Studio sessions with Azure Cloud compute environments, you must attach Seqera credentials with an Entra client ID/client secret pair. These credentials must also include your Azure subscription ID and Storage account configuration. See [Register an application in Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app) and [Add and manage application credentials in Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/identity-platform/how-to-add-credentials?tabs=client-secret) for more information. ### Required permissions For granular control over the permissions granted to Seqera, use [Azure custom roles](https://learn.microsoft.com/en-us/azure/role-based-access-control/custom-roles) and [assign](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal) them to the service principal. The full role JSON definition is: ```json { "properties": { "roleName": "seqera-azure-cloud", "description": "Role assumed by Seqera Platform to create Azure Cloud compute environments", "assignableScopes": [ "/subscriptions/" ], "permissions": [ { "actions": [ "Microsoft.Compute/virtualMachines/read", "Microsoft.Compute/virtualMachines/write", "Microsoft.Compute/virtualMachines/delete", "Microsoft.Compute/virtualMachines/deallocate/action", "Microsoft.Compute/virtualMachines/attachDetachDataDisks/action", "Microsoft.Resources/subscriptions/resourceGroups/write", "Microsoft.Resources/subscriptions/resourceGroups/read", "Microsoft.Resources/subscriptions/resourceGroups/delete", "Microsoft.Network/virtualNetworks/read", "Microsoft.Network/virtualNetworks/write", "Microsoft.Network/virtualNetworks/delete", "Microsoft.Network/virtualNetworks/subnets/read", "Microsoft.Network/virtualNetworks/subnets/write", "Microsoft.Network/virtualNetworks/subnets/delete", "Microsoft.Network/virtualNetworks/subnets/join/action", "Microsoft.Network/networkInterfaces/delete", "Microsoft.Network/networkInterfaces/write", "Microsoft.Network/networkInterfaces/read", "Microsoft.Network/networkInterfaces/join/action", "Microsoft.ManagedIdentity/userAssignedIdentities/read", "Microsoft.ManagedIdentity/userAssignedIdentities/write", "Microsoft.ManagedIdentity/userAssignedIdentities/delete", "Microsoft.ManagedIdentity/userAssignedIdentities/assign/action", "Microsoft.Authorization/roleAssignments/read", "Microsoft.Authorization/roleAssignments/write", "Microsoft.Authorization/roleAssignments/delete", "Microsoft.Authorization/roleDefinitions/read", "Microsoft.Authorization/roleDefinitions/write", "Microsoft.Authorization/roleDefinitions/delete", "Microsoft.Insights/DataCollectionRules/Read", "Microsoft.Insights/DataCollectionRules/Write", "Microsoft.Insights/DataCollectionRules/Delete", "Microsoft.Insights/DataCollectionEndpoints/Write", "Microsoft.Insights/DataCollectionEndpoints/Delete", "Microsoft.OperationalInsights/workspaces/write", "Microsoft.OperationalInsights/workspaces/read", "Microsoft.OperationalInsights/workspaces/delete", "Microsoft.OperationalInsights/workspaces/sharedkeys/action", "Microsoft.OperationalInsights/workspaces/tables/read", "Microsoft.OperationalInsights/workspaces/tables/write", "Microsoft.OperationalInsights/workspaces/tables/delete", "Microsoft.OperationalInsights/workspaces/query/read", "Microsoft.OperationalInsights/workspaces/query/Tables.Custom/read", "Microsoft.Storage/storageAccounts/blobServices/containers/read", "Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action" ], "notActions": [], "dataActions": [ "Microsoft.Insights/Telemetry/Write", "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read", "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/tags/read", "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write", "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/tags/write", "Microsoft.OperationalInsights/workspaces/tables/data/read" ], "notDataActions": [] } ] } } ``` See [Start from JSON](https://learn.microsoft.com/en-us/azure/role-based-access-control/custom-roles-portal#start-from-json) to create this custom role in the Azure Portal. This role definition can be applied as-is for convenience, or it can be broken down into smaller roles. The purpose for each permission is outlined in the following sections. #### Compute environment creation The following permissions are required to provision resources in the Azure account when first creating the compute environment: ```json { "properties": { "roleName": "seqera-azure-cloud-create", "description": "Role assumed by Seqera Platform to create Azure Cloud compute environments", "assignableScopes": [ "/subscriptions/" ], "permissions": [ { "actions": [ "Microsoft.Resources/subscriptions/resourceGroups/read", "Microsoft.Resources/subscriptions/resourceGroups/write", "Microsoft.Storage/storageAccounts/blobServices/containers/read", "Microsoft.Network/virtualNetworks/read", "Microsoft.Network/virtualNetworks/write", "Microsoft.Network/virtualNetworks/subnets/read", "Microsoft.Network/virtualNetworks/subnets/write", "Microsoft.ManagedIdentity/userAssignedIdentities/read", "Microsoft.ManagedIdentity/userAssignedIdentities/write", "Microsoft.Authorization/roleAssignments/read", "Microsoft.Authorization/roleAssignments/write", "Microsoft.Authorization/roleDefinitions/read", "Microsoft.Authorization/roleDefinitions/write", "Microsoft.Insights/DataCollectionRules/Read", "Microsoft.Insights/DataCollectionRules/Write", "Microsoft.Insights/DataCollectionEndpoints/Write", "Microsoft.OperationalInsights/workspaces/read", "Microsoft.OperationalInsights/workspaces/write", "Microsoft.OperationalInsights/workspaces/tables/write" ], "notActions": [], "dataActions": [], "notDataActions": [] } ] } } ``` #### Pipeline and Studio launch The following permissions are required to launch pipelines and Studios: ```json { "properties": { "roleName": "seqera-azure-cloud-launch", "description": "Role assumed by Seqera Platform to launch Studios and pipelines on Azure Cloud compute environments", "assignableScopes": [ "/subscriptions/" ], "permissions": [ { "actions": [ "Microsoft.Compute/virtualMachines/read", "Microsoft.Compute/virtualMachines/write", "Microsoft.Compute/virtualMachines/delete", "Microsoft.Compute/virtualMachines/deallocate/action", "Microsoft.Compute/virtualMachines/attachDetachDataDisks/action", "Microsoft.Network/networkInterfaces/read", "Microsoft.Network/networkInterfaces/write", "Microsoft.Network/networkInterfaces/join/action", "Microsoft.Network/virtualNetworks/subnets/join/action", "Microsoft.ManagedIdentity/userAssignedIdentities/assign/action", "Microsoft.Insights/DataCollectionRules/Write", "Microsoft.Insights/DataCollectionEndpoints/Write" ], "notActions": [], "dataActions": [ "Microsoft.Insights/Telemetry/Write" ], "notDataActions": [] } ] } } ``` #### Live stream log fetching The following permissions are required to fetch logs for the pipeline execution while the task is running: ``` json { "properties": { "roleName": "seqera-azure-cloud-logs", "description": "Role to be assumed by Seqera Platform to read live-streamed logs for Azure Cloud compute environments", "assignableScopes": [ "/subscriptions/" ], "permissions": [ { "actions": [ "Microsoft.OperationalInsights/workspaces/query/read", "Microsoft.OperationalInsights/workspaces/query/Tables.Custom/read" ], "notActions": [], "dataActions": [ "Microsoft.OperationalInsights/workspaces/tables/data/read" ], "notDataActions": [] } ] } } ``` #### Data-links The following permissions are required to work with [Data Explorer](../data/data-explorer) data-links on Azure: ```json { "properties": { "roleName": "seqera-azure-cloud-data-links", "description": "Role assumed by Seqera Platform to access data-links in Azure Cloud compute environments", "assignableScopes": [ "/subscriptions/" ], "permissions": [ { "actions": [ "Microsoft.Storage/storageAccounts/blobServices/containers/read", "Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action" ], "notActions": [], "dataActions": [ "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read", "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/tags/read", "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write", "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/tags/write" ], "notDataActions": [] } ] } } ``` #### Compute environment termination and resource disposal The following permissions are required to delete the resources created for the compute environment: ```json { "properties": { "roleName": "seqera-azure-cloud-dispose", "description": "Role assumed by Seqera Platform to delete Azure Cloud compute environment resources", "assignableScopes": [ "/subscriptions/" ], "permissions": [ { "actions": [ "Microsoft.Resources/subscriptions/resourceGroups/delete", "Microsoft.Network/virtualNetworks/delete", "Microsoft.Network/virtualNetworks/subnets/delete", "Microsoft.Network/networkInterfaces/delete", "Microsoft.ManagedIdentity/userAssignedIdentities/delete", "Microsoft.Authorization/roleAssignments/delete", "Microsoft.Authorization/roleDefinitions/delete", "Microsoft.Insights/DataCollectionRules/Delete", "Microsoft.Insights/DataCollectionEndpoints/Delete", "Microsoft.OperationalInsights/workspaces/delete", "Microsoft.OperationalInsights/workspaces/tables/delete" ], "notActions": [], "dataActions": [], "notDataActions": [] } ] } } ``` ## Add Azure Cloud credentials ### Create a custom role in Microsoft Entra First, you must create a custom role with the permissions required for Seqera to manage Azure resources. 1. Save the relevant permissions from the preceding sections to a local JSON file. Replace `` in the `assignableScopes` field of each permission with your Azure subscription ID. 1. In the Azure Portal, go to **Subscriptions** and select your subscription. 1. To create a custom role, select **Access control (IAM)**, then **Add** in the **Create a custom role** section. 1. Provide the following details: - **Custom role name**: e.g., `seqera-azure-cloud` - **Description**: e.g., `Role for Seqera Platform to manage Azure Cloud compute environments` - **Baseline permissions**: Select **Start from JSON** - **File**: Select the local JSON file you saved earlier. 1. Select **Next** and review the permissions to ensure all have been included correctly. 1. Select **Next** and confirm that the assignable scope is your subscription ID. 1. Select **Next**. If you found errors in the previous step, you can edit the JSON file here. 1. Select **Next** and then **Create** to save the role. ### Register an application in Microsoft Entra ID Create an application for Seqera to use for authentication: 1. In the Azure Portal, go to **App registrations** and select **New registration**. 1. Give the app a descriptive name, such as `SeqeraPlatformApp`. 1. Select `Single tenant` for the supported account types. 1. Create a client secret for the application. Seqera will use this value to authenticate to Azure, so keep it secret and store it securely. 1. Under **Certificates & secrets**, select **New client secret** and give it a description such as `SeqeraPlatformSecret`. Set the expiration to a duration that matches your security policy. Select **Add**. After registration, you'll be taken to the application overview page. Copy and save the following values: - **Application (client) ID**: This is your Client ID - **Directory (tenant) ID**: This is your Tenant ID ### Assign the custom role to the service principal Grant the service principal the necessary permissions by assigning the custom role. 1. In the Azure Portal, navigate to **Subscriptions** and select your subscription. Then select **Access control (IAM)** and **Add role assignment** in the **Grant access to this resource** section. 1. Select the **Privileged administrator roles** tab and select the role you created earlier, then select **Next**. 1. Choose **Select members** and search for the application name (`SeqeraPlatformApp`). Then choose **Select**, then **Next**. 1. Select **Review + assign**, **Review**, and then **Assign**. 1. Under **What user can do**, select **Allow user to assign all roles except privileged administrator roles Owner, UAA, RBAC (Recommended)**, then select **Next**. 1. Check the final details and select **Review + assign**. ### Configure Seqera Platform credentials Add the service principal credentials to Seqera: 1. Sign in to your Seqera workspace and navigate to the **Credentials** tab. 1. Select **Add credentials**, select **Azure** as the provider, and select the **Cloud** tab for Microsoft Entra ID authentication. 1. Enter the details of the credentials you saved earlier: - **Name**: Provide a descriptive name, such as `AzureCloudCredentials` - **Subscription ID**: Your Azure subscription ID - **Tenant ID**: Your Directory (tenant) ID from the [Register an application in Microsoft Entra ID](#register-an-application-in-microsoft-entra-id) instructions - **Client ID**: Your Application (client) ID from the [Register an application in Microsoft Entra ID](#register-an-application-in-microsoft-entra-id) instructions - **Client secret**: Your client secret value from the [Register an application in Microsoft Entra ID](#register-an-application-in-microsoft-entra-id) instructions - **Blob Storage account name**: Your Azure Storage account name 1. Review the details, then select **Add** to save the credentials. ### Create a compute environment Create a compute environment in Seqera using the credentials: 1. In your Seqera workspace, navigate to the **Compute Environments** tab and select **Add Compute Environment**. 1. Select **Azure Cloud** as the target platform. 1. From the **Credentials** drop-down, select the credentials you created previously. 1. Enter a name for the compute environment. 1. Enter or select a **Location** for the compute environment. 1. Select the **Work directory** as the Azure blob container you plan to use as the Nextflow working directory. The container must be in the same **Location** as selected in the previous step. 1. (Optional) Under **Advanced options**, specify an **Instance Type**. If left blank, the default virtual machine used is a `Standard_D2ds_v4`. 1. Select **Create** to save the compute environment. ## Advanced options - (Optional) **Subscription ID**: The ID of the subscription where resources must be deployed. If not specified, the subscription ID of the credentials is used. - **Instance Type**: The virtual machine type used by the compute environment. Choosing the instance type will directly allocate the CPU and memory available for computation. See [virtual machine sizes](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/overview) for a comprehensive list of instance types and their resource limitations. - **Boot disk size**: The size of the OS disk (in GB) for the virtual machines launched by this compute environment, including Studio session VMs. Must be between 50 and 4095 GB. If undefined, the OS disk uses the default size of the VM image. - **Virtual network**: An existing Azure virtual network (VNet) in the configured location. The drop-down is populated with VNets discovered in your Azure account for the selected location. When specified, Platform uses this network for all VMs launched in this compute environment and skips network provisioning. Leave blank to let Platform provision a dedicated VNet automatically. :::note The VNet must exist in the same location as the compute environment. Specifying a VNet that does not exist in the location, or a subnet that does not belong to the selected VNet, causes compute environment creation to fail. ::: - **Subnets**: One or more subnet addresses within the selected VNet. VMs are placed in the first listed subnet at launch time. Leave blank to use the first available subnet on the VNet. This field has no effect when no VNet is specified. Any [network security groups (NSGs)](https://learn.microsoft.com/en-us/azure/virtual-network/network-security-groups-overview) attached to the selected subnet are applied to VMs launched in this compute environment. --- ## Amazon EKS [Amazon EKS](https://aws.amazon.com/eks/) provides managed Kubernetes clusters that enable the execution of containerized workloads at scale. Seqera Platform offers native support for Amazon EKS clusters as Compute Environments for Nextflow pipelines. ## Requirements - Seqera Platform needs an IAM User to obtain details about the EKS cluster, and to fetch log files from the S3 bucket, if one is used as work directory. This user must have the permissions detailed in the [Required Platform IAM permissions](#required-platform-iam-permissions) section. Optionally, permissions can instead be granted to an IAM role that the IAM user can assume when accessing AWS resources. - To use Fusion (recommended) to access data hosted on S3, including writing files to the Nextflow work directory, you need an IAM role that allows the EKS Service Account that Seqera pods use to interact with AWS resources. Refer to section [Configure EKS Service Account IAM role for Fusion v2](#configure-eks-service-account-iam-role-for-fusion-v2) for details. Create a separate IAM role from the optional one assumed by the IAM user to separate the permissions needed by the EKS Service Account from those needed by the IAM user. If you plan to use legacy storage instead of Fusion, you can skip this step. :::tip Seqera Platform assumes an EKS cluster already exists. Follow the [cluster preparation](./k8s) instructions to create the resources required by Seqera. Some administrative privileges are also needed to allow the IAM User to access the cluster, as detailed in the [EKS access](#allow-an-iam-user-or-role-access-to-eks) section. ::: Once you meet all the prerequisites, configure an [Amazon EKS Compute Environment](#amazon-eks-compute-environment) in Seqera. ## Required Platform IAM permissions Seqera Platform requires an IAM user with specific permissions to launch pipelines, explore buckets with Data Explorer, and run Studio sessions on the AWS EKS compute environment. Some permissions are mandatory for the compute environment to function correctly, while others are optional and enable features like populating drop-down lists in the Platform UI. Attach permissions directly to an [IAM user](#iam-user-creation), or to an [IAM role](#iam-role-creation-optional) that the IAM user can assume. A permissive and broad policy with all the required permissions is provided here for a quick start. However, we recommend following the principle of least privilege and only granting the necessary permissions for your use case, as shown in the following sections.
Full permissive policy (for reference) ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "EKSClusterAccessCanBeRestricted", "Effect": "Allow", "Action": [ "eks:DescribeCluster", "eks:ListClusters" ], "Resource": "*" }, { "Sid": "OptionalS3PlatformDataAccessCanBeRestricted", "Effect": "Allow", "Action": [ "s3:Get*", "s3:List*", "s3:PutObject", "s3:PutObjectTagging", "s3:DeleteObject" ], "Resource": "*" } ] } ```
### EKS cluster access Seqera needs permissions to list EKS clusters in the selected region and to describe the selected cluster to retrieve its connection details. The `eks:ListClusters` action cannot be restricted to specific resources, but the `eks:DescribeCluster` action can be restricted to the specific cluster used as compute environment. ```json { "Sid": "EKSClusterListing", "Effect": "Allow", "Action": [ "eks:ListClusters" ], "Resource": "*" }, { "Sid": "EKSClusterDescription", "Effect": "Allow", "Action": [ "eks:DescribeCluster" ], "Resource": "arn:aws:eks:::cluster/" } ``` No other permissions are required for the IAM user to launch pipelines on the EKS compute environment, as the Service Account created in the [cluster preparation](./k8s) phase performs the actual management of pods and resources, which the IAM user can access via EKS authentication, detailed [in the EKS access section below](#allow-an-iam-user-or-role-access-to-eks). ### S3 access (optional) Seqera automatically attempts to fetch a list of S3 buckets available in the AWS account connected to Platform, to provide them in a drop-down to be used as Nextflow working directory, and make the compute environment creation smoother. This feature is optional, and users can type the bucket name manually when setting up a compute environment. To allow Seqera to fetch the list of buckets in the account, the `s3:ListAllMyBuckets` action can be added, and it must have the `Resource` field set to `*`. The `s3:ListAllMyBuckets` action also allows Data Explorer to auto-discover the data repositories accessible to your workspace credentials. Seqera offers several products to manipulate data on AWS S3 buckets, such as [Studios](../studios/overview) and [Data Explorer](../data/data-explorer); if these features are not needed the related permissions can be omitted. The IAM policy can be scoped down to only allow limited read/write permissions in certain S3 buckets used by Studios/Data Explorer. For each bucket you want to browse, upload to, or download from with Data Explorer, grant `s3:GetObject` and `s3:PutObject` on the bucket objects, and `s3:ListBucket`, `s3:GetBucketLocation`, `s3:GetBucketPolicy`, and `s3:GetBucketAcl` on the bucket itself. In addition, the policy must include permission to check the region and list the content of the S3 bucket used as Nextflow work directory. We also recommend granting the `s3:GetObject` permission on the work directory path to fetch Nextflow log files. :::note If you opted to create a separate S3 bucket only for Nextflow work directories, the IAM user or the Role it assumes only need read access to it. The IAM role used by the EKS Service Account (detailed in the [separate section](#configure-eks-service-account-iam-role-for-fusion-v2)) must have full read/write access to the work directory bucket to allow Fusion to operate correctly. ::: ```json { "Sid": "S3CheckBucketWorkDirectory", "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::example-bucket-used-as-work-directory" ] }, { "Sid": "S3ReadOnlyNextflowLogFiles", "Effect": "Allow", "Action": [ "s3:GetObject" ], "Resource": [ "arn:aws:s3:::example-bucket-used-as-work-directory/path/to/work/directory/*" ] }, { "Sid": "S3ReadWriteBucketsForStudiosDataExplorer", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectTagging", "s3:GetBucketLocation", "s3:GetBucketPolicy", "s3:GetBucketAcl", "s3:ListBucket", "s3:PutObject", "s3:PutObjectTagging", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::example-bucket-read-write-studios", "arn:aws:s3:::example-bucket-read-write-studios/*", "arn:aws:s3:::example-bucket-read-write-data-explorer", "arn:aws:s3:::example-bucket-read-write-data-explorer/*" ] } ``` :::note `s3:GetBucketLocation` allows Data Explorer to resolve each bucket's region. `s3:GetBucketPolicy` and `s3:GetBucketAcl` allow it to inspect each bucket's access configuration when it lists and connects to data repositories. If you prefer not to enumerate individual actions, the `s3:Get*` and `s3:List*` wildcards shown in the full permissive policy also cover these actions. ::: ## Create the IAM policy The policy above must be created in the AWS account where the EKS and S3 resources are located. 1. Open the [AWS IAM console](https://console.aws.amazon.com/iam). 1. From the left navigation menu, select **Policies** under **Access management**. 1. Select **Create policy**. 1. On the **Policy editor** section, select the **JSON** tab. 1. Following the instructions detailed in the [IAM permissions breakdown section](#required-platform-iam-permissions) replace the default text in the policy editor area under the **JSON** tab with a policy adapted to your use case, then select **Next**. 1. Enter a name and description for the policy on the **Review and create** page, then select **Create policy**. ## IAM user creation Seqera requires an Identity and Access Management (IAM) User to describe EKS clusters and S3 buckets in your AWS account. We recommend creating a separate IAM policy rather an IAM User inline policy, as the latter only allows 2048 characters, which may not be sufficient for all the required permissions. In certain scenarios, for example when multiple users need to access the same AWS account, an IAM role with the required permissions can be created instead, and the IAM user allowed to assume the role, as detailed in the [IAM role creation (optional)](#iam-role-creation-optional) section. ### Create an IAM user 1. From the [AWS IAM console](https://console.aws.amazon.com/iam), select **Users** in the left navigation menu, then select **Create User** at the top right of the page. 1. Enter a name for your user (e.g., _seqera_) and select **Next**. 1. Under **Permission options**, select **Attach policies directly**, then search for and select the policy created above, and select **Next**. * If you instead prefer to make the IAM user assume a role to manage AWS resources (see the [IAM role creation (optional)](#iam-role-creation-optional) section), create a policy with the following content (edit the AWS principal with the ARN of the role created) and attach it to the IAM user: ```json { "Sid": "AssumeRole", "Effect": "Allow", "Action": "sts:AssumeRole", "Resource": "arn:aws:iam:::role/", "Condition": { "StringEquals": { "sts:ExternalId": "" } } } ``` 1. On the last page, review the user details and select **Create user**. The user has now been created. For more details see the [AWS documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html). ### Obtain IAM user credentials To get the credentials needed to connect Seqera to your AWS account, follow these steps: 1. From the [AWS IAM console](https://console.aws.amazon.com/iam), select **Users** in the left navigation menu, then select the newly created user from the users table. 1. Select the **Security credentials** tab, then select **Create access key** under the **Access keys** section. 1. In the **Use case** dialog that appears, select **Command line interface (CLI)**, then tick the confirmation checkbox at the bottom to acknowledge that you want to proceed creating an access key, and select **Next**. 1. Optionally provide a description for the access key, like the reason for creating it, then select **Create access key**. 1. Save the **Access key** and **Secret access key** in a secure location as they are needed when configuring credentials in Seqera. ## IAM role creation (optional) Rather than attaching permissions directly to the IAM user, you can create an IAM role with the required permissions and allow the IAM user to assume that role when accessing AWS resources. This is useful when multiple IAM users are used to access the same AWS account: this way the actual permissions to operate on the resources are only granted to a single centralized role. 1. From the [AWS IAM console](https://console.aws.amazon.com/iam), select **Roles** in the left navigation menu, then select **Create role** at the top right of the page. 1. Select **Custom trust policy** as the type of trusted entity, provide the following policy and edit the AWS principal with the ARN of the IAM user created in the [IAM user creation](#iam-user-creation) section, then select **Next**. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam:::user/" ] }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } }, { "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam:::user/" ] }, "Action": "sts:TagSession" } ] } ``` 1. On the **Permissions** page, search for and select the policy created in the [IAM user creation](#iam-user-creation) section, then select **Next**. 1. Give the role a name and optionally a description, review the details of the role, optionally provide tags to help you identify the role, then select **Create role**. Multiple users can be specified in the trust policy by adding more ARNs to the `Principal` section. :::note Seqera Platform generates the `External ID` value during AWS credential creation. For role-based credentials, use this exact value in your IAM trust policy (`sts:ExternalId`). ::: ### Role-based trust policy example (Seqera Enterprise) For role-based AWS credentials in Enterprise, use the AWS IAM role configured in your deployment (``) in your trust policy and enforce the `External ID` generated during credential creation: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } }, { "Effect": "Allow", "Principal": { "AWS": "" }, "Action": "sts:TagSession" } ] } ``` :::info In Seqera Enterprise, a jump role is optional. If you configure one, use your own jump role ARN as the trusted principal in the trust policy. The **Assume role** value in the credential form is the customer IAM role ARN in your AWS account. It is separate from any optional jump role configuration. ::: :::info To use role-based access with no External ID, set `TOWER_ALLOW_INSTANCE_CREDENTIALS=true` in your deployment [configuration](../enterprise/configuration/overview#compute-environments). Then create AWS credentials using an IAM role ARN only (no access key, secret key, or External ID), and remove the entire `Condition` block for `sts:ExternalId` from your trust policy. ::: ## AWS credential options AWS credentials can be configured in three ways: - **Key-based credentials**: Access key and secret key with direct IAM permissions. If you provide a role ARN in **Assume role**, the **Generate External ID** switch is displayed and External ID generation is optional. - **Role-based credentials (recommended)**: Use role assumption only (no static keys). Paste the IAM role ARN which Seqera must use for accessing your AWS resources in **Assume role**. External ID is generated automatically when you save. - **Workload identity federation**: No stored credential. Platform exchanges a short-lived OpenID Connect (OIDC) token for temporary credentials against an IAM role you control, and stores only the role Amazon Resource Name (ARN). See [Workload identity federation][wif] for the trust policy, permission policies, and setup steps. Use the IAM role ARN which Seqera must use for accessing your AWS resources in **Assume role**. This field is available for both key-based and role-based credentials. It is optional for key-based credentials and required for role-based credentials. Existing credentials created before March 2026 continue to work without changes. `TOWER_ALLOW_INSTANCE_CREDENTIALS=true` configuration behavior remains unchanged. ## Configure EKS Service Account IAM role for Fusion v2 To use [Fusion v2](https://docs.seqera.io/fusion) in your Amazon EKS compute environment, an AWS S3 bucket must be used as work directory and both the head and compute Service Accounts (if separate) must have access to the S3 bucket specified as the work directory. If you do not plan to use Fusion in favor of legacy storage, you can skip this section. 1. Create an IAM role with the following permissions: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::" ] }, { "Action": [ "s3:GetObject", "s3:PutObject", "s3:PutObjectTagging", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::/*" ], "Effect": "Allow" } ] } ``` Replace `` with the bucket name used as work directory. 1. The IAM role must also have a trust relationship with the Kubernetes service account (or accounts) that Seqera uses to manage the EKS cluster, which is `tower-launcher-sa` in the default configuration.: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam:::oidc-provider/oidc.eks..amazonaws.com/id/" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks..amazonaws.com/id/:aud": "sts.amazonaws.com", "oidc.eks..amazonaws.com/id/:sub": "system:serviceaccount::" } } } ] } ``` Replace ``, ``, ``, ``, `` with the corresponding values. 1. Annotate the Kubernetes Service Account with the IAM role: ```shell kubectl annotate serviceaccount --namespace eks.amazonaws.com/role-arn=arn:aws:iam:::role/ ``` Replace `` (by default `tower-launcher-sa`, as created in the [cluster preparation guide](./k8s)), ``, and `` with the corresponding values previously defined. This will allow pods using that service account to assume the IAM role and access the S3 bucket specified as work directory. See the [AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html) for further details. ## Allow an IAM User or Role access to EKS Configure the EKS cluster to allow the IAM user (or the IAM role it assumes) to access the cluster and manage pods. 1. Retrieve from the [AWS IAM console](https://console.aws.amazon.com/iam) the ARN of the [IAM User](#iam-user-creation) or [IAM Role](#iam-role-creation-optional) previously created. :::note The AWS credentials for the IAM user will be used in the Seqera compute environment configuration. ::: 1. Modify the EKS aws-auth ConfigMap to allow the IAM User to access the cluster and manage pods. This step may require cluster administrator privileges: ```bash kubectl edit configmap -n kube-system aws-auth ``` 1. In the editor that opens, edit the `mapUsers` section to add the following entry, replacing `` with the user ARN retrieved from the AWS IAM console: ```yaml mapUsers: | - userarn: username: tower-launcher-user groups: - tower-launcher-role ``` Alternatively, an IAM role can be allowed to authenticate to the cluster: in this case, the role ARN must be specified in the **Assume role** field when configuring the Seqera compute environment (step 9 in the [Amazon EKS compute environment](#amazon-eks-compute-environment) section), the role must have a trust relationship with the Seqera IAM user, and the role `` must be added to the `mapRoles` section of the EKS auth configuration instead: ```yaml mapRoles: | - rolearn: username: tower-launcher-role groups: - tower-launcher-role ``` See the [AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/auth-configmap.html) for more details on modifying the aws-auth ConfigMap of an EKS cluster. ## Amazon EKS compute environment :::caution Your compute environment uses resources that you may be charged for in your AWS account. See [Cloud costs](../monitoring/cloud-costs) for guidelines to manage cloud resources effectively and prevent unexpected costs. ::: Once all prerequisites are met, create a Seqera EKS compute environment: 1. Select **Compute environments** from the navigation menu of the Seqera Workspace where you want to setup the CE. 1. Enter a descriptive name for this environment, e.g., `Amazon EKS (eu-west-1)`. 1. Select **Amazon EKS** as the target platform. 1. Under **Storage**, select either **Fusion storage** (recommended) or **Legacy storage**. The [Fusion v2](https://docs.seqera.io/fusion) virtual distributed file system allows access to your AWS S3-hosted data (`s3://` URLs). This eliminates the need to configure a shared file system in your Kubernetes cluster. See [Configure EKS Service Account IAM role for Fusion v2](#configure-eks-service-account-iam-role-for-fusion-v2) below. 1. From the **Credentials** drop-down, select existing AWS credentials, or select **+** to add new credentials. If you're using existing credentials, skip to step 9. The user must have the IAM permissions required to describe and list EKS clusters, per Service Account role requirements. :::note You can create multiple credentials in your Seqera environment. See [Credentials](../credentials/overview). ::: 1. Enter a name, e.g., `EKS Credentials`. 1. Under **AWS credential mode**, select **Keys** or **Role**. 1. For **Keys** mode: - Add the **Access key** and **Secret key** obtained from the AWS IAM console. - Optionally paste the IAM role ARN which Seqera must use for accessing your AWS resources in **Assume role**. - If you paste a role ARN in **Assume role**, the **Generate External ID** switch is displayed. Generating an External ID is optional in **Keys** mode. - If **Generate External ID** is selected, an External ID is automatically generated and shown after you save the credential. 1. For **Role** mode: - Paste the IAM role ARN which Seqera must use for accessing your AWS resources in **Assume role**. - External ID is generated automatically when you save the credential. :::note When using AWS keys without an assumed role, the associated AWS user must have been granted permissions to operate on the cloud resources directly. When an assumed role is provided, the IAM user keys are only used to retrieve temporary credentials impersonating the role specified: this could be useful when e.g. multiple IAM users are used to access the same AWS account, and the actual permissions to operate on the resources are only granted to the role. ::: 1. Select a **Region**, e.g., `eu-west-1 - Europe (Ireland)`. If using Fusion v2, this region must match the location of the S3 bucket you plan to use as work directory. 1. Select a **Cluster name** from the list of available EKS clusters in the selected region. 1. Specify the **Namespace** created in the [cluster preparation](./k8s) instructions, `tower-nf` by default. 1. Specify the **Head service account** created in the [cluster preparation](./k8s) instructions, `tower-launcher-sa` by default. :::note If you enable Fusion v2 (**Fusion storage** in step 4 above), the head service account must have access to the S3 storage bucket specified as your work directory. In the [Advanced options](#amazon-eks-advanced-options) below, a service account for compute jobs need to also be specified to allow pods to interact with AWS. ::: 1. Define the **Work directory** used as the working directory by Nextflow pipelines. If using Fusion v2, this must be an S3 bucket (e.g., `s3://my-bucket/work-dir`). If using Legacy storage, this must the name of a Persistent Volume Claim (PVC) created in the [cluster preparation](./k8s) instructions, e.g., `tower-scratch`. 1. Apply [**Resource labels**](../resource-labels/overview) to the cloud resources produced by this compute environment. Workspace default resource labels are prefilled. 1. Expand **Staging options** to include: - Optional [pre- or post-run Bash scripts](../launch/advanced#pre-and-post-run-scripts) that execute before or after the Nextflow pipeline execution in your environment. - Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the **Nextflow config file** field in the pipeline launch form. These values can be overridden during pipeline launch. Here's an example configuration to require the compute pods to be scheduled on specific nodes: ```groovy k8s { pod = [ [ nodeSelector: 'myNodeSelector=my-nodes-for-k8s-as-compute' ], [ toleration: [ key: 'myNodeSelector', operator: 'Equal', value: 'my-nodes-for-k8s-as-compute', effect: 'NoSchedule' ] ] ] } ``` :::info Configuration settings in this field override the same values in the pipeline repository `nextflow.config` file. See [Nextflow config file](../launch/advanced#nextflow-config-file) for more information on configuration priority. ::: 1. Under **Environment variables**, add each variable with a **Name**, **Value**, and **Target Environment**: - **Head job**: Adds the variable to the Nextflow head job container, which evaluates `nextflow.config` and submits tasks to the compute backend. Use this target for variables that Nextflow or its plugins read, such as `NXF_OPTS`, `NXF_JVM_ARGS`, `NXF_PLUGINS_DEFAULT`, or proxy settings the head node uses to reach external services. - **Compute job**: Adds the variable to the worker containers that run individual pipeline tasks. Use this target for variables your pipeline tools read, such as `OPENAI_API_KEY` for a process that calls the OpenAI API, registry credentials needed inside the task container, or tool-specific settings like `JAVA_HOME`. - **Head and Compute jobs**: Adds the variable to both the head job and the compute jobs. Use this target for values needed in both places, such as an HTTP proxy used by both Nextflow and task tools, or a credential needed in both the head job and individual compute tasks. :::note For sensitive values such as API keys and tokens, use [pipeline secrets](../secrets/overview) instead of custom environment variables. Custom environment variables are stored in the compute environment configuration and cannot be edited after creation. To rotate a value, recreate the compute environment. ::: 1. Configure any advanced options described in the next section, as needed. ### Amazon EKS advanced options Amazon EKS compute environments include advanced options for storage and work directory paths, resource allocation, and pod customization. - The **Storage mount path** is the file system path where Seqera mounts the Storage claim (default: `/scratch`). - The **Work directory** is the file system path that Nextflow pipelines use as a working directory. This must be the storage mount path (default) or a subdirectory of it. - The **Compute service account** is the service account that Nextflow uses to submit tasks (default: the `default` account in the given namespace). :::note If you enable Fusion v2 (**Fusion storage** in step 4 above), the compute service account must have access to the S3 storage bucket specified as your work directory. This can be the same Service Account used by the Head jobs (`tower-launcher-sa`, created in the [cluster preparation](./k8s) guide), or a separate Service Account with more granular permissions. ::: - The **Pod cleanup policy** determines when Seqera deletes the run's Nextflow head pod. **On success** (the default) deletes it when the run succeeds, **Always** deletes it regardless of outcome, and **Never** keeps it. This policy does not delete task pods. See [Kubernetes pods remain after a run ends](../troubleshooting_and_faqs/nextflow#kubernetes-pods-remain-after-a-run-ends). - Use **Custom head pod specs** to provide custom options for the Nextflow workflow pod (e.g., `nodeSelector`, `affinity`, etc). For example: ```yaml spec: nodeSelector: disktype: ssd ``` - Use **Head job CPUs** and **Head job memory** to specify resource requirements of the Nextflow workflow pods. :::info See [Launch pipelines](../launch/launchpad) to start executing workflows in your Amazon EKS compute environment. ::: [wif]: ../credentials/workload_identity --- ## Google Kubernetes Engine [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) is a managed Kubernetes cluster that allows the execution of containerized workloads in Google Cloud at scale. Seqera Platform offers native support for GKE clusters to streamline the deployment of Nextflow pipelines. ## Requirements See [here](../compute-envs/google-cloud-batch#configure-google-cloud) for instructions to set up your Google Cloud account and other services (such as Cloud storage). You must have a GKE cluster up and running. Follow the [cluster preparation](../compute-envs/k8s#cluster-preparation) instructions to create the resources required by Seqera. In addition to the generic Kubernetes instructions, you must make a number of modifications specific to GKE. ### Service account role You must grant cluster access to the service account used by the Seqera compute environment. To do this, update the [service account _RoleBinding_](https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control#rolebinding): ```yaml cat << EOF | kubectl apply -f - --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: tower-launcher-userbind subjects: - kind: User name: apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: tower-launcher-role apiGroup: rbac.authorization.k8s.io --- EOF ``` Replace `` with the corresponding service account, e.g., `test-account@test-project-123456.google.com.iam.gserviceaccount.com`. See [Role-based access control](https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control) for more information. ## Seqera compute environment :::caution Your Seqera compute environment uses resources that you may be charged for in your Google Cloud account. See [Cloud costs](../monitoring/cloud-costs) for guidelines to manage cloud resources effectively and prevent unexpected costs. ::: After you've prepared your Kubernetes cluster and granted cluster access to your service account, create a Seqera GKE compute environment: 1. In a Seqera workspace, select **Compute environments > New environment**. 1. Enter a descriptive name for this environment, e.g., _Google Kubernetes Engine (europe-west1)_. 1. From the **Provider** drop-down, select **Google Kubernetes Engine**. 1. Under **Storage**, select either **Fusion storage** (recommended) or **Legacy storage**. The [Fusion v2](https://docs.seqera.io/fusion) virtual distributed file system allows access to your Google Cloud-hosted data (`gs://` URLs). This eliminates the need to configure a shared file system in your Kubernetes cluster. See [Fusion v2](#fusion-v2) below. 1. From the **Credentials** drop-down, select existing GKE credentials, or select **+** to add new credentials. If you choose to use existing credentials, skip to step 8. 1. Enter a name for the credentials, e.g., _GKE Credentials_. 1. Enter the **Service account key** for your Google service account. :::tip You can create multiple credentials in your Seqera environment. See [Credentials](../credentials/overview). ::: 1. Select the **Location** of your GKE cluster. :::caution GKE clusters can be either regional or zonal. For example, `us-west1` identifies the United States West-Coast _region_, which has three _zones_: `us-west1-a`, `us-west1-b`, and `us-west1-c`. Seqera Platform's auto-completion only shows regions. You should manually edit this field if you're using a zonal GKE cluster. ::: 1. Select or enter the **Cluster name** of your GKE cluster. 1. Specify the **Namespace** created in the [cluster preparation](../compute-envs/k8s#cluster-preparation) instructions. This is _tower-nf_ by default. 1. Specify the **Head service account** created in the [cluster preparation](../compute-envs/k8s#cluster-preparation) instructions. This is _tower-launcher-sa_ by default. :::note If you enable Fusion v2 (**Fusion storage** in step 4 above), the head service account must have access to the Google Cloud storage bucket specified as your work directory. ::: 1. Specify the **Storage claim** created in the [cluster preparation](../compute-envs/k8s#cluster-preparation) instructions. This serves as a scratch filesystem for Nextflow pipelines. The storage claim is called _tower-scratch_ in the provided examples. :::note The **Storage claim** isn't needed when Fusion v2 is enabled. ::: 1. Apply [**Resource labels**](../resource-labels/overview) to the cloud resources consumed by this compute environment. Workspace default resource labels are prefilled. 1. Expand **Staging options** to include: - Optional [pre- or post-run Bash scripts](../launch/advanced#pre-and-post-run-scripts) that execute before or after the Nextflow pipeline execution in your environment. - Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the **Nextflow config file** field in the pipeline launch form. These values can be overridden during pipeline launch. :::info Configuration settings in this field override the same values in the pipeline repository `nextflow.config` file. See [Nextflow config file](../launch/advanced#nextflow-config-file) for more information on configuration priority. ::: 1. Under **Environment variables**, add each variable with a **Name**, **Value**, and **Target Environment**: - **Head job**: Adds the variable to the Nextflow head job container, which evaluates `nextflow.config` and submits tasks to the compute backend. Use this target for variables that Nextflow or its plugins read, such as `NXF_OPTS`, `NXF_JVM_ARGS`, `NXF_PLUGINS_DEFAULT`, or proxy settings the head node uses to reach external services. - **Compute job**: Adds the variable to the worker containers that run individual pipeline tasks. Use this target for variables your pipeline tools read, such as `OPENAI_API_KEY` for a process that calls the OpenAI API, registry credentials needed inside the task container, or tool-specific settings like `JAVA_HOME`. - **Head and Compute jobs**: Adds the variable to both the head job and the compute jobs. Use this target for values needed in both places, such as an HTTP proxy used by both Nextflow and task tools, or a credential needed in both the head job and individual compute tasks. :::note For sensitive values such as API keys and tokens, use [pipeline secrets](../secrets/overview) instead of custom environment variables. Custom environment variables are stored in the compute environment configuration and cannot be edited after creation. To rotate a value, recreate the compute environment. ::: 1. Configure any advanced options described in the next section, as needed. 1. Select **Create** to finalize the compute environment setup. ### Advanced options Seqera Platform compute environments for GKE include advanced options for storage and work directory paths, resource allocation, and pod customization. - The **Storage mount path** is the file system path where the Storage claim is mounted (default: `/scratch`). - The **Work directory** is the file system path used as a working directory by Nextflow pipelines. It must be the storage mount path (default) or a subdirectory of it. - The **Compute service account** is the service account used by Nextflow to submit tasks (default: the `default` account in the given namespace). - The **Pod cleanup policy** determines when Seqera deletes the run's Nextflow head pod. **On success** (the default) deletes it when the run succeeds, **Always** deletes it regardless of outcome, and **Never** keeps it. This policy does not delete task pods. See [Kubernetes pods remain after a run ends](../troubleshooting_and_faqs/nextflow#kubernetes-pods-remain-after-a-run-ends). - Use **Custom head pod specs** to provide custom options for the Nextflow workflow pod (`nodeSelector`, `affinity`, etc). For example: ```yaml spec: nodeSelector: disktype: ssd ``` - Use **Custom service pod specs** to provide custom options for the compute environment pod. See above for an example. - Use **Head job CPUs** and **Head job memory** to specify the hardware resources allocated for the Nextflow workflow pod. :::info See [Launch pipelines](../launch/launchpad) to start executing workflows in your GKE compute environment. ::: ### Fusion v2 To use [Fusion v2](https://docs.seqera.io/fusion) in your Seqera GKE compute environment: 1. Use Seqera Platform version 23.1 or later. 1. Use an S3 bucket as the pipeline work directory. 1. Both the head service and compute service accounts must have access to the Google Cloud storage bucket specified as the work directory.
Configure IAM to use Fusion v2 1. Ensure the **Workload Identity** feature is enabled for the cluster: - **Enable Workload Identity** in the cluster **Security** settings. - **Enable GKE Metadata Server** in the node group **Security** settings. 1. Allow the IAM service account access to your Google storage bucket: ```shell gcloud storage buckets add-iam-policy-binding gs:// --role roles/storage.objectAdmin --member serviceAccount:@.iam.gserviceaccount.com ``` The role must have at least `storage.objects.create`, `storage.objects.get`, and `storage.objects.list` permissions. 1. Allow the Kubernetes service account to impersonate the IAM service account: ```shell gcloud iam service-accounts add-iam-policy-binding @.iam.gserviceaccount.com --role roles/iam.workloadIdentityUser --member "serviceAccount:.svc.id.goog[/]" ``` 1. Annotate the Kubernetes service account with the email address of the IAM service account: ```shell kubectl annotate serviceaccount --namespace iam.gke.io/gcp-service-account=@.iam.gserviceaccount.com ``` See the [GKE documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to) for further details.
--- ## Google Cloud Batch :::note This guide assumes you have an existing Google Cloud account. Sign up for a free account [here](https://cloud.google.com/). Seqera Platform provides integration to Google Cloud via the [Batch API](https://cloud.google.com/batch/docs/reference/rest). ::: The guide is split into two parts: 1. How to configure your Google Cloud account to use the Batch API. 2. How to create a Google Cloud Batch compute environment in Seqera. ## Configure Google Cloud ### Create a project Go to the [Google Project Selector page](https://console.cloud.google.com/projectselector2) and select an existing project, or select **Create project**. Enter a name for your new project, e.g., *tower-nf*. If you are part of an organization, the location defaults to your organization. ### Enable billing See [Enable, disable, or change billing for a project](https://cloud.google.com/billing/docs/how-to/modify-project) to enable billing in your Google Cloud account. ### Enable APIs See [Enable API wizard](https://console.cloud.google.com/flows/enableapi?apiid=batch.googleapis.com%2Ccompute.googleapis.com%2Cstorage-api.googleapis.com) to enable the following APIs for your project: * Batch API * Compute Engine API * Cloud Storage API Select your project from the drop-down and select **Enable**. Alternatively, enable each API manually by selecting your project in the navigation bar and visiting each API page: * [Batch API](https://console.cloud.google.com/marketplace/product/google/batch.googleapis.com) * [Compute Engine API](https://console.cloud.google.com/marketplace/product/google/compute.googleapis.com) * [Cloud Storage API](https://console.cloud.google.com/marketplace/product/google/storage-api.googleapis.com) ### IAM Seqera requires a service account with appropriate permissions to interact with your Google Cloud resources. As an IAM user, you must have access to the service account that submits Batch jobs. :::caution By default, Google Cloud Batch uses the default Compute Engine service account to submit jobs. This service account is granted the Editor (`roles/Editor`) role. While this service account has the necessary permissions needed by Seqera, this role is not recommended for production environments. Control job access using a custom service account with only the permissions necessary for Seqera to execute Batch jobs instead. ::: #### Service account permissions [Create a custom service account](https://cloud.google.com/iam/docs/service-accounts-create#creating) with at least the following permissions: * Batch Agent Reporter (`roles/batch.agentReporter`) on the project * Batch Job Editor (`roles/batch.jobsEditor`) on the project * Logs Writer (`roles/logging.logWriter`) on the project (to let jobs generate logs in Cloud Logging) * Service Account User (`roles/iam.serviceAccountUser`) * Service Usage Consumer (`roles/serviceusage.serviceUsageConsumer`) * Secret Manager Secret Accessor (`roles/secretmanager.secretAccessor`) on the project (required if your pipelines use Seqera secrets, because the head job and tasks read secrets from GCP Secret Manager) * Storage Bucket Viewer (`roles/storage.bucketViewer`) on the project (required if you grant Storage access per bucket instead of project-wide Storage Admin, which already includes the `storage.buckets.list` permission) If your Google Cloud project does not require access restrictions on any of its Cloud Storage buckets, you can grant project Storage Admin (`roles/storage.admin`) permissions to your service account to simplify setup. To grant access only to specific buckets, add the service account as a principal on each bucket individually. See [Cloud Storage bucket](#cloud-storage-bucket) below. Seqera needs the `storage.buckets.list` permission at the project level to list buckets when you create a compute environment, to browse buckets in Data Explorer, and to [validate the credential](./preflight-checks). Bucket-level grants cannot confer `storage.buckets.list`. #### User permissions Ask your Google Cloud administrator to grant you the following IAM user permissions to interact with your custom service account: * Batch Job Editor (`roles/batch.jobsEditor`) on the project * Service Account User (`roles/iam.serviceAccountUser`) on the job's service account (default: Compute Engine service account) * View Service Accounts (`roles/iam.serviceAccountViewer`) on the project #### Authentication methods Seqera supports two methods for authenticating with Google Cloud: **Service account keys** To authenticate using a service account key, create a [service account JSON key file](https://cloud.google.com/iam/docs/keys-list-get#get-key): 1. In the Google Cloud navigation menu, select **IAM & Admin > Service Accounts**. 2. Select the email address of the service account. :::note The Compute Engine default service account is not recommended for production environments due to its powerful permissions. To use a service account other than the Compute Engine default, specify the service account email address under **Advanced options** on the Seqera compute environment creation form. ::: 3. Select **Keys > Add key > Create new key**. 4. Select **JSON** as the key type. 5. Select **Create**. A JSON file is downloaded to your computer. This file contains the credential needed to configure the compute environment in Seqera. You can manage your key from the **Service Accounts** page. **Workload identity federation** Workload identity federation is the recommended authentication method for production and regulated environments because it eliminates long-lived service account keys. Seqera Platform authenticates to Google Cloud with short-lived OIDC tokens instead. Before you create the credential, configure the workload identity pool, provider, attribute mapping, and service account impersonation in Google Cloud. See [Workload identity federation][wif] for the setup steps, the subject model, and cloud audit attribution. ### Cloud Storage bucket Google Cloud Storage is a type of **object storage**. To access files and store the results for your pipelines, create a **Cloud bucket** that your Seqera service account can access. #### Create a Cloud Storage bucket 1. In the hamburger menu (**≡**), select **Cloud Storage**. 2. From the **Buckets** tab, select **Create**. 3. Enter a name for your bucket. You reference this name when you create the compute environment in Seqera. 4. Select **Region** for the **Location type** and select the **Location** for your bucket. Use this location when you create the compute environment in Seqera. :::note The Batch API is available in a limited number of [locations](https://cloud.google.com/batch/docs/locations). These locations are only used to store metadata about the pipeline operations. The storage bucket and compute resources can be in any region. ::: 5. Select **Standard** for the default storage class. 6. To restrict public access to your bucket data, select the **Enforce public access prevention on this bucket** checkbox. 7. Under **Access control**, select **Uniform**. 8. Select any additional object data protection tools, per your organization's data protection requirements. 9. Select **Create**. #### Assign bucket permissions 1. After the bucket is created, you are redirected to the **Bucket details** page. 2. Select **Permissions**, then **Grant access** under **View by principals**. 3. Copy the email address of your service account into **New principals**. 4. Select the **Storage Admin** role, then select **Save**. :::tip You've created a project, enabled the necessary Google APIs, created a bucket, and created credentials for your service account. You now have what you need to set up a new compute environment in Seqera. ::: ### Seqera compute environment :::caution Your Seqera compute environment uses resources that incur charges in your Google Cloud account. See [Cloud costs](https://docs.seqera.io/platform-enterprise/monitoring/cloud-costs) for guidelines to manage cloud resources effectively and prevent unexpected costs. ::: After you create your Google Cloud resources, create a Seqera Platform compute environment: 1. In a workspace, select **Compute Environments > New Environment**. 2. Enter a descriptive name for this environment, e.g., *Google Cloud Batch (europe-north1)*. 3. Select **Google Cloud Batch** as the target platform. #### Credentials 1. From the **Credentials** drop-down, select existing Google credentials or select **+** to add new credentials. If you choose to use existing credentials, skip to the next section. 2. Enter a name for the credentials, e.g., *Google Cloud Credentials*. 3. Select the credential type: - **Service account key**: Paste the contents of the JSON key file created in the [service account keys](#authentication-methods) section. - **Workload Identity**: Enter the **Service Account Email**, **Workload Identity Provider** path, and optionally the **Token Audience** as described in the [workload identity federation](#authentication-methods) section. #### Location and work directory Select the **Location** where you execute your pipelines. See [Location](https://cloud.google.com/compute/docs/regions-zones#available) to learn more. In the **Pipeline work directory** field, enter your storage bucket URL. For example, `gs://my-bucket`. This bucket must be accessible in the location selected in the previous step. :::note When you specify a Cloud Storage bucket as your work directory, this bucket is used for the Nextflow [cloud cache](https://docs.seqera.io/nextflow/cache-and-resume#cache-stores) by default. You can specify an alternative cache location with the **Nextflow config file** field on the pipeline [launch](../launch/launchpad#launch-form) form. ::: #### Seqera features - Select **Enable Wave containers** to facilitate access to private container repositories and provision containers in your pipelines using the Wave containers service. See [Wave containers](https://docs.seqera.io/nextflow/wave) for more information. - Select **Enable Fusion v2** to allow access to your Google Cloud Storage data via the [Fusion v2](https://docs.seqera.io/fusion) virtual distributed file system. This speeds up most data operations. The Fusion v2 file system requires Wave containers to be enabled. See [Fusion file system](../supported_software/fusion/overview) for configuration details. :::note The compute recommendations below are based on internal benchmarking performed by Seqera. Benchmark runs of [nf-core/rnaseq](https://github.com/nf-core/rnaseq) used profile `test_full`, consisting of an input dataset with 16 FASTQ files and a total size of approximately 123.5 GB. ::: 1. Use Seqera Platform version 23.1 or later. 2. Use a Google Cloud Storage bucket as the pipeline work directory. 3. Enable **Wave containers** and **Fusion v2**. 4. Specify suitable virtual machine types and local storage settings, or accept the default machine settings listed below. Use an `n2-highmem-16-lssd` VM or larger for production. :::note To specify virtual machine settings in Platform during compute environment creation, use the **Global Nextflow config** field to apply custom Nextflow process directives to all pipeline runs launched with this compute environment. To specify virtual machine settings per pipeline run in Platform, or as a persistent configuration in your Nextflow pipeline repository, use Nextflow process directives. See [Google Cloud Batch process definition](https://docs.seqera.io/nextflow/google#process-definition) for more information. ::: When Fusion v2 is enabled, Seqera Platform applies the following virtual machine settings: * Unless you specify an instance template, Nextflow requests a 375 GB scratch disk for all compute jobs. Families that support local SSDs use `local-ssd`. Other families use a persistent disk or Hyperdisk volume. * If you do not specify a machine type, Seqera Platform selects a VM from families that support local SSDs. * Local SSDs are only offered in multiples of 375 GB. Increment the scratch disk per process with the `disk` directive, for example `disk = [request: 750.GB, type: 'local-ssd']`. Without the `type` option, `disk` sets the boot disk size instead. To work with files larger than 100 GB, use at least two local SSDs (750 GB or more). * Fusion v2 can also use persistent disks for caching. See [Scratch disk](https://docs.seqera.io/fusion/guide/gcp-batch#scratch-disk) to choose a disk type. * Instance templates override the `disk` directive. To use Fusion with an instance template, the template must include a `local-ssd` disk named `fusion` with 375 GB. * Use the `machineType` directive to specify a VM instance type, family, or custom machine type in a comma-separated list of patterns. For example, `c2-*`, `n1-standard-1`, `custom-2-4`, `n*`, `m?-standard-*`. :::note Wave containers and Fusion v2 are recommended features for added capability and improved performance, but neither are required to execute workflows in your compute environment. ::: #### GCP resources Enable **Spot** to use Spot instances, which have significantly reduced cost compared to On-Demand instances. :::note From Nextflow version 24.10, the default Spot reclamation retry setting changed to `0` on AWS and Google. By default, no internal retries are attempted on these platforms. Spot reclamations now lead to an immediate failure, exposed to Nextflow in the same way as other generic failures (for example, returning `exit code 1` on AWS). Nextflow treats these failures like any other job failure unless you configure a retry strategy. For more information, see [Spot instance failures and retries](../troubleshooting_and_faqs/nextflow#spot-instance-failures-and-retries). Selecting the 'enable Fusion snapshots' option (Google Cloud Batch) changes the default Spot reclamation retry setting to `5`. ::: :::info When a Spot instance is reclaimed by Google Cloud, Seqera Platform displays a human-readable description in the task details. Google Batch reserves exit codes in the 50001–59999 range for infrastructure events: | Exit code | Description | |-----------|-------------| | 50001 | Spot instance was reclaimed by Google Cloud | | 50002 | VM became unresponsive (host event or crash) | | 50003 | VM unexpectedly rebooted during task execution | | 50004 | Task reached unresponsive time limit and could not be cancelled | | 50005 | Task exceeded maximum allowed runtime | Exit codes 50006–59999 display a generic infrastructure failure message. Standard application exit codes (1–255) are displayed as before. ::: Apply [**Resource labels**](../resource-labels/overview) to the cloud resources consumed by this compute environment. Workspace default resource labels are prefilled. #### Scripting and environment variables * Expand **Staging options** to include: + Optional [pre- or post-run Bash scripts](../launch/advanced#pre-and-post-run-scripts) that execute before or after the Nextflow pipeline execution in your environment. + Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the **Nextflow config file** field in the pipeline launch form. These values can be overridden during pipeline launch. :::info Configuration settings in this field override the same values in the pipeline repository `nextflow.config` file. See [Nextflow config file](../launch/advanced#nextflow-config-file) for more information on configuration priority. ::: * Under **Environment variables**, add each variable with a **Name**, **Value**, and **Target Environment**: - **Head job**: Adds the variable to the Nextflow head job container, which evaluates `nextflow.config` and submits tasks to the compute backend. Use this target for variables that Nextflow or its plugins read, such as `NXF_OPTS`, `NXF_JVM_ARGS`, `NXF_PLUGINS_DEFAULT`, or proxy settings the head node uses to reach external services. - **Compute job**: Adds the variable to the worker containers that run individual pipeline tasks. Use this target for variables your pipeline tools read, such as `OPENAI_API_KEY` for a process that calls the OpenAI API, registry credentials needed inside the task container, or tool-specific settings like `JAVA_HOME`. - **Head and Compute jobs**: Adds the variable to both the head job and the compute jobs. Use this target for values needed in both places, such as an HTTP proxy used by both Nextflow and task tools, or a credential needed in both the head job and individual compute tasks. :::note For sensitive values such as API keys and tokens, use [pipeline secrets](../secrets/overview) instead of custom environment variables. Custom environment variables are stored in the compute environment configuration and cannot be edited after creation. To rotate a value, recreate the compute environment. ::: #### Advanced options :::note If you use VM instance templates for the head or compute jobs (see step 8 below), resource allocation and networking values in the templates override any conflicting values you set while creating your Seqera compute environment. ::: 1. Enable **Use Private Address** to ensure that your Google Cloud VMs aren't accessible to the public internet. :::note This option requires both a **VPC** and a **Subnet** (step 7). You cannot create the compute environment without them. ::: 2. Use **Boot disk size** to control the persistent disk size that each task and the head job are provided. 3. Use **Boot Disk Image** to select a specific boot disk image for the compute instances. The drop-down is populated with available images from the GCP Compute API and supports autocomplete filtering. This field is optional. If not set, Google Batch uses the default image. 4. Use **Instance Type** to select one or more machine types for the compute instances. The drop-down is populated with available instance types for the selected region and supports autocomplete filtering. You can select multiple specific instance types or use family wildcards (for example, `c2-*` or `n*`) to allow Google Batch to choose from a family. This field is optional. If not set, Google Batch automatically selects an appropriate machine type. :::note The **Instance Type** field sets the default machine type selection at the compute environment level. You can override this for individual processes using the `machineType` [process directive](https://docs.seqera.io/nextflow/google#process-definition) in your Nextflow configuration, which accepts a comma-separated list of patterns (for example, `c2-*`, `n1-standard-1`, `custom-2-4`). ::: 5. Use **Head job CPUs** and **Head job memory** to specify the CPUs and memory allocated for the head job. :::caution The default head job resource values are insufficient for production pipelines. The Nextflow head job is a JVM process that tracks every submitted task, manages pipeline state, and polls the GCP Batch API. If the head job runs out of memory mid-run, the pipeline fails. Tasks already running on worker VMs run to completion, but no new tasks are scheduled. Output files that were already written are not cleaned up automatically. Results may be incomplete. Size the head job based on the number of tasks in your pipeline: | Pipeline scale | Tasks | Recommended CPUs | Recommended memory | |---|---|---|---| | Small | Up to 100 | 2 | 4 GB | | Medium | 100–500 | 4 | 8 GB | | Large | 500+ | 8 | 16 GB | Head job memory scales with the number of concurrent tasks and total pipeline duration. Long-running pipelines keep thousands of task records in memory for resumability, and need more memory than short pipelines with the same peak parallelism. Increase CPUs if task scheduling is slow or the head job logs show high garbage collection (GC) pressure. For large pipelines, you can also increase the JVM heap directly by setting `NXF_JVM_ARGS="-Xms4g -Xmx12g"` as a **Head job** environment variable (see [Scripting and environment variables](#scripting-and-environment-variables)). ::: :::note If you specify a **Head job instance template** (see step 9), the template's machine type overrides the **Head job CPUs** and **Head job memory** values set here. ::: 6. Use **Service Account email** to specify a service account email address other than the Compute Engine default to execute workflows with this compute environment (recommended for production environments). 7. Use **VPC** and **Subnet** to specify the name of a VPC network and subnet to be used by this compute environment. You can apply network tags directly in the **Network Tags** field (see below) or through VM instance templates used for the Nextflow head and compute jobs. :::note You must specify both a **VPC** and **Subnet** for your compute environment to use either. ::: 8. Use **Network Tags** to apply GCP network tags to the compute instances in this environment. Network tags control which firewall rules and routing policies apply to your instances within a VPC. Enter tags as free-text values. Tags must follow GCP format requirements: lowercase letters, numbers, and hyphens only, between 1 and 63 characters. You can add up to 64 tags per instance. :::note Network tags require a **VPC** and **Subnet** to be configured. This field is disabled when no VPC is set. ::: 9. Use **Head job instance template** and **Compute jobs instance template** to specify the name or fully-qualified reference of a VM instance template, without the `template://` prefix, to use for the head and compute jobs. [VM instance templates](https://cloud.google.com/compute/docs/instance-templates) allow you to define the resources allocated to Batch jobs. Configuration values defined in a VM instance template override any conflicting values you specify while creating your Seqera compute environment. :::caution Seqera does not validate the VM instance template you specify in these fields. Generally, use templates that define only the machine type, network, disk, and configuration values that will not change across multiple VM instances and Seqera compute environments. See [Create instance templates](https://cloud.google.com/compute/docs/instance-templates/create-instance-templates) for instructions to create your instance templates. ::: To prevent errors during workflow execution, ensure that the instance templates you use are suitably configured for your needs with an appropriate machine type. You can define multiple instance templates with varying machine type sizes in your Nextflow configuration using the `machineType` [process directive](https://docs.seqera.io/nextflow/google#process-definition) (e.g., `process.machineType = 'template://my-template-name'`). You can use [process selectors](https://docs.seqera.io/nextflow/config#config-process-selectors) to assign separate templates to each of your processes. Select **Create** to finalize the compute environment setup. :::info See [Launch pipelines](https://docs.seqera.io/platform-enterprise/launch/launchpad) to start executing workflows in your Google Cloud Batch compute environment. ::: [wif]: ../credentials/workload_identity --- ## Google Cloud :::note This compute environment type is currently in public preview. Consult this guide for the latest information on recommended configuration and limitations. This guide assumes you already have a GCP account with a valid subscription. ::: Many of the current implementations of compute environments for cloud providers rely on the use of batch services such as AWS Batch, Azure Batch, and Google Batch for the execution and management of submitted jobs, including pipelines and Studio session environments. Batch services are suitable for large-scale workloads, but they add management complexity. In practical terms, the currently used batch services result in some limitations: - **Long launch delay**: When you launch a pipeline or Studio in a batch compute environment, there's a delay of several minutes before the pipeline or Studio session environment is in a running state. This is caused by the batch services that need to provision the associated compute service to run a single job. - **Complex setup**: Standard batch services require complex identity management policies and configuration of multiple components including batch job definitions, task specifications, resource policies, etc. The Google Cloud compute environment addresses these pain points with: - **Faster startup time**: By eliminating the per-task overhead of VM provisioning, environment bootstrapping, and container image pulling that occurs with traditional batch, Nextflow pipelines reach a `Running` status and Studio sessions connect in under a minute (a 4x improvement compared to classic GCP Batch compute environments). - **Simplified configuration**: Fewer configurable options, with opinionated defaults, provide the best Nextflow pipeline and Studio session execution environment, with both Wave and Fusion enabled. - **Fewer GCP dependencies**: Direct use of Compute Engine eliminates the reliance on Google Batch APIs and reduces the required IAM permissions to core services (Compute Engine, Cloud Storage, and IAM), resulting in a simpler architecture with fewer potential points of failure. This type of compute environment is best suited to run Studios and small to medium-sized pipelines. It offers more predictable compute pricing, given the fixed instance types. It spins up a standalone Google Compute Engine instance and executes a Nextflow pipeline or Studio session with a local executor on the Google Compute Engine machine. At the end of the execution, the instance is terminated. ## Limitations The Nextflow pipeline will run entirely on a single Google Compute Engine instance. If the instance does not have sufficient resources, the pipeline execution will fail. For this reason, the number of tasks Nextflow can execute in parallel is limited by the number of cores of the instance type selected. If you need more computing resources, you must create a new compute environment with a larger instance type. This makes the compute environment less suited for larger, more complex pipelines. ## Supported locations The following locations are currently supported: - `asia-east1` - `asia-east2` - `asia-northeast1` - `asia-northeast2` - `asia-northeast3` - `asia-south1` - `asia-south2` - `asia-southeast1` - `asia-southeast2` - `australia-southeast1` - `australia-southeast2` - `europe-central2` - `europe-north1` - `europe-southwest1` - `europe-west1` - `europe-west2` - `europe-west3` - `europe-west4` - `europe-west6` - `europe-west8` - `europe-west9` - `europe-west10` - `europe-west12` - `me-central1` - `me-west1` - `northamerica-northeast1` - `northamerica-northeast2` - `southamerica-east1` - `southamerica-west1` - `us-central1` - `us-east1` - `us-east4` - `us-east5` - `us-south1` - `us-west1` - `us-west2` - `us-west3` - `us-west4` ## Networking A Google Cloud compute environment launches a Compute Engine instance into a VPC network and subnet. - **Outbound access is required**: When the compute environment is configured to launch VMs without an external IP address, the VPC must provide outbound access through Cloud NAT and Private Google Access. - **Firewall rules target network tags**: Network tags configured on the compute environment are applied to launched VMs, and are what your VPC firewall rules target. - **Studios sessions are outbound-only**: The Seqera Connect client inside a Studio session opens a tunnel outward to the Connect server, and all session traffic — including SSH when enabled — travels over that outbound connection. **No inbound path to the session VM is required**, so you do not need ingress firewall rules or source-IP allow-lists for dynamically launched Studio VMs. The network does need outbound access to the Connect server. See [Networking](../studios/overview#networking) in the Studios documentation. For the ports and directions to configure on your firewall, see [Firewall configuration](../enterprise/advanced-topics/firewall-configuration). ## Requirements ### Platform credentials To create and launch pipelines or Studio sessions with this compute environment type, you must attach Seqera credentials for the cloud provider. Some permissions are mandatory for the compute environment to be created and function correctly; others are used to pre-fill Platform options, which are optional. This compute environment supports workload identity federation, which authenticates Seqera Platform to Google Cloud with short-lived OIDC tokens instead of a long-lived service account key. Before you create the credential, configure the workload identity pool, provider, attribute mapping, and service account impersonation in Google Cloud. See [Workload identity federation][wif] for the setup steps, the subject model, and cloud audit attribution. ### Required permissions #### Service account permissions​ [Create a custom service account](https://cloud.google.com/iam/docs/service-accounts-create#creating) with at least the following permissions: - Compute instance admin (`roles/compute.instanceAdmin.v1`) - Project IAM admin (`roles/resourcemanager.projectIamAdmin`) - Service Account Admin (`roles/iam.serviceAccountAdmin`) - Service Account User (`roles/iam.serviceAccountUser`) - Service Usage Consumer (`roles/serviceusage.serviceUsageConsumer`) If your Google Cloud project does not require access restrictions on any of its Cloud Storage buckets, you can grant project Storage Admin (`roles/storage.admin`) permissions to your service account to simplify setup. To grant access only to specific buckets, add the service account as a principal [on each bucket individually](https://docs.seqera.io/platform-cloud/compute-envs/google-cloud-batch#cloud-storage-bucket). For each Google Cloud compute environment created in the Seqera platform, a separate service account is created with the necessary permissions to launch pipelines/studios. ## Advanced options - **Use an ARM64 architecture instance**: Select this option to enable an ARM architecture instance to be created for your compute workload. This option defaults to using a [C4A machine series](https://cloud.google.com/compute/docs/general-purpose-machines#c4a_series) VM with Google's ARM-based Axion™ processor. - **User GPU-enabled instance**: Select this option to enable a GPU-enabled instance to be created for your compute workload. This option defaults to using an [A2 machine series](https://cloud.google.com/compute/docs/gpus) VM with an NVIDIA A100 GPU. - **Instance type**: The Compute Engine machine type used by the compute environment. Choosing the instance type will directly allocate the CPU and memory available for computation. See the [machine resource type documentation](https://cloud.google.com/compute/docs/machine-resource) for a comprehensive list of instance types and their resource limitations. :::note It is not possible to specify instance templates with predefined machine types, storage, bootstrapped, etc. ::: - **Image**: The image defining the operating system and pre-installed software for the VM. Currently only [Ubuntu LTS](https://cloud.google.com/compute/docs/images/os-details#ubuntu_lts) Google public image project images are available and supported. For GPU-enabled instances, a Deep Learning VM base image with CUDA pre-installed is automatically selected (See [Google Deep Learning VM Images](https://cloud.google.com/deep-learning-vm/docs/images#base_versions) for more details). Optimized, Seqera-owned custom images will be available in a future release. - **Boot disk size**: The size of the boot disk for the Compute Engine instance. A standard persistent disk (`pd-standard`) is used. If undefined, a default 50 GB volume will be used. - **Zone**: The [zone](https://cloud.google.com/compute/docs/regions-zones) within the selected region where the VM will be provisioned (defaults to the first zone in the alphabetical list). [wif]: ../credentials/workload_identity --- ## HPC compute environments Seqera Platform streamlines the deployment of Nextflow pipelines into both cloud-based and on-prem HPC clusters and supports compute environment creation for the following management and scheduling solutions: - [Altair PBS Pro](https://www.altair.com/pbs-professional/) - [Grid Engine](https://www.altair.com/grid-engine/) - [IBM Spectrum LSF](https://www.ibm.com/products/hpc-workload-management/details) (Load Sharing Facility) - [Moab](http://docs.adaptivecomputing.com/suite/8-0/basic/help.htm#topics/moabWorkloadManager/topics/intro/productOverview.htm) - [Slurm](https://slurm.schedmd.com/overview.html) ## Requirements To launch pipelines into an **HPC** cluster from Seqera, the following requirements must be satisfied: - The cluster should allow outbound connections to the Seqera web service. - The cluster queue used to run the Nextflow head job must be able to submit cluster jobs. - The Nextflow runtime version **21.02.0-edge** (or later) must be installed on the cluster. ## Credentials Seqera requires SSH access to your HPC cluster to run pipelines. Use [managed identities](../credentials/managed_identities) to enable granular access control and preserve individual cluster user identities. You can also use workspace [SSH credentials](../credentials/ssh_credentials) for cluster login, but this provides service account access to your HPC to all Platform users. This means that all users will be granted the same file system access, and all activity is logged under the same user account on your HPC cluster. For HPC clusters that do not allow direct access through an SSH client, a secure connection can be authenticated with [Tower Agent](../supported_software/agent/overview). ## Work and launch directories For instances where the work directory or launch directory must be set dynamically at runtime, you can use variable expansion. This works in conjunction with Tower Agent. The path that results from variable expansion must exist before workflow execution as the agent does not create directories. For example, if the HPC cluster file system has a `/workspace` directory with subdirectories for each user that can run jobs, the value for the work directory can be the following: `/workspace/$TW_AGENT_USER`. For a user `user1`, the work directory resolves to the `/workspace/user1` directory. The following variables are supported: - `TW_AGENT_WORKDIR`: Resolves to the work directory for Tower Agent. By default, this directory resolves to the `${HOME}/work` path, where `HOME` is the home directory of the user that the agent runs as. The work directory can be overridden by specifying the `--work-dir` argument when configuring Tower Agent. For more information, see the [Tower Agent][agent] documentation. - `TW_AGENT_USER`: Resolves to the username that the agent is running as. By default, this is the Unix username that the agent runs as. On systems where the agent cannot determine which user it runs as, it falls back to the value of the `USER` environment variable. ## HPC compute environment To create a new **HPC** compute environment: 1. In a Seqera workspace, select **Compute environments > New environment**. 1. Enter a descriptive name for this environment. Use only alphanumeric characters, dashes, and underscores. 1. Select your HPC environment from the **Platform** drop-down. 1. Select your existing managed identity, SSH, or Tower Agent credentials, or select **+** and **SSH** or **Tower Agent** to add new credentials. 1. Enter the absolute path of the **Work directory** to be used on the cluster. You can use the `TW_AGENT_WORKDIR` and `TW_AGENT_USER` variables in the file system path. :::caution All managed identity users must be a part of the same Linux user group. The group must have access to the HPC compute environment work directory. Set group permissions for the work directory as follows (replace `sharedgroupname` and `` with your group name and work directory): ```bash chgrp -R sharedgroupname chmod -R g+wxs setfacl -Rdm g::rwX ``` These commands change the group ownership of all files and directories in the work directory to `sharedgroupname`, ensure new files inherit the directory's group, and apply default ACL entries to allow the group read, write, and execute permissions for new files and directories. This setup facilitates shared access and consistent permissions management in the directory. ::: 1. Enter the absolute path of the **Launch directory** to be used on the cluster. If omitted, it will be the same as the work directory. 1. Enter the **Login hostname**. This is usually the hostname or public IP address of the cluster's login node. 1. Enter the **Head queue name**. This is the [default](https://docs.seqera.io/nextflow/process#queue) cluster queue to which the Nextflow job will be submitted. 1. Enter the **Compute queue name**. This is the [default](https://docs.seqera.io/nextflow/process#queue) cluster queue to which the Nextflow job will submit tasks. 1. Expand **Staging options** to include: - Optional [pre- or post-run Bash scripts](../launch/advanced#pre-and-post-run-scripts) that execute before or after the Nextflow pipeline execution in your environment. - Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the **Nextflow config file** field in the pipeline launch form. These values can be overridden during pipeline launch. :::info Configuration settings in this field override the same values in the pipeline repository `nextflow.config` file. See [Nextflow config file](../launch/advanced#nextflow-config-file) for more information on configuration priority. ::: 1. Under **Environment variables**, add each variable with a **Name**, **Value**, and **Target Environment**: - **Head job**: Adds the variable to the Nextflow head job container, which evaluates `nextflow.config` and submits tasks to the compute backend. Use this target for variables that Nextflow or its plugins read, such as `NXF_OPTS`, `NXF_JVM_ARGS`, `NXF_PLUGINS_DEFAULT`, or proxy settings the head node uses to reach external services. - **Compute job**: Adds the variable to the worker containers that run individual pipeline tasks. Use this target for variables your pipeline tools read, such as `OPENAI_API_KEY` for a process that calls the OpenAI API, registry credentials needed inside the task container, or tool-specific settings like `JAVA_HOME`. - **Head and Compute jobs**: Adds the variable to both the head job and the compute jobs. Use this target for values needed in both places, such as an HTTP proxy used by both Nextflow and task tools, or a credential needed in both the head job and individual compute tasks. :::note For sensitive values such as API keys and tokens, use [pipeline secrets](../secrets/overview) instead of custom environment variables. Custom environment variables are stored in the compute environment configuration and cannot be edited after creation. To rotate a value, recreate the compute environment. ::: 1. Configure any advanced options needed: - Use the **Nextflow queue size** to limit the number of jobs that Nextflow can submit to the scheduler at the same time. - Use the **Head job submit options** to add platform-specific submit options for the head job. You can optionally apply these options to compute jobs as well: :::note Once set during compute environment creation, these options can't be overridden at pipeline launch time. ::: :::note In IBM LSF compute environments, use **Unit for memory limits**, **Per job memory limits**, and **Per task reserve** to control how memory is requested for Nextflow jobs. ::: 1. Select **Create** to finalize the creation of the compute environment. See [Launch pipelines](../launch/launchpad) to start executing workflows in your HPC compute environment. [agent]: ../supported_software/agent/overview --- ## Kubernetes [Kubernetes](https://kubernetes.io/) is the leading technology for the deployment and orchestration of containerized workloads in cloud-native environments. Seqera Platform streamlines the deployment of Nextflow pipelines into Kubernetes, both for cloud-based and on-prem clusters. The following instructions create a Seqera compute environment for a **generic Kubernetes** distribution. See [Amazon EKS](./eks) or [Google Kubernetes Engine (GKE)](./gke) for EKS and GKE compute environment instructions. ## Cluster preparation To prepare your Kubernetes cluster for the deployment of Nextflow pipelines using Seqera, this guide assumes that you've already created the cluster and that you have administrative privileges. This guide applies a Kubernetes manifest that creates a service account named `tower-launcher-sa` and the associated role bindings, all contained in the `tower-nf` namespace. Seqera uses the service account to launch Nextflow pipelines. Use this service account name when setting up the compute environment for this Kubernetes cluster in Seqera. **Prepare your Kubernetes cluster for Seqera Platform** 1. Verify the connection to your Kubernetes cluster: ```bash kubectl cluster-info ``` 1. Create a file named `tower-launcher.yml` with the following YAML: ```yaml file=../_templates/k8s/tower-launcher.yml showLineNumbers ``` 1. Apply the manifest: ```bash kubectl apply -f tower-launcher.yml ``` 1. Create a persistent API token for the `tower-launcher-sa` service account: ```bash kubectl apply -f - < ``` ## Seqera compute environment After you've prepared your Kubernetes cluster for Seqera integration, create a compute environment: **Create a Seqera Kubernetes compute environment** 1. In a workspace, select **Compute environments > New environment**. 1. Enter a descriptive name for this environment, e.g., _K8s cluster_. 1. Select **Kubernetes** as the target platform. 1. From the **Credentials** drop-down, select existing Kubernetes credentials, or select **+** to add new credentials. If you choose to use existing credentials, skip to step 7. :::tip You can create multiple credentials in your Seqera workspace. See [Credentials](../credentials/overview). ::: 1. Enter a name, such as _K8s Credentials_. 1. Select either the **Service Account Token** or **X509 Client Certs** tab: - To authenticate using a Kubernetes service account, enter your **Service account token**. Obtain the token with the following command: ```bash kubectl -n tower-nf describe secret | grep -E '^token' | cut -f2 -d':' | tr -d '\t ' ``` Replace `` with the name of the service account token created in the [cluster preparation](#cluster-preparation) instructions (default: `tower-launcher-token`). - To authenticate using an X509 client certificate, paste the contents of your certificate and key file (including the `-----BEGIN...-----` and `-----END...-----` lines) in the **Client certificate** and **Client Key** fields respectively. See the [Kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/certificates/) for instructions to generate your client certificate and key. 1. Enter the **Control plane URL**, obtained with this command: ```bash kubectl cluster-info ``` It can also be found in your `~/.kube/config` file under the `server` field corresponding to your cluster. 1. Specify the **SSL certificate** to authenticate your connection. Find the certificate data in your `~/.kube/config` file. It is the `certificate-authority-data` field corresponding to your cluster. 1. Specify the **Namespace** created in the [cluster preparation](#cluster-preparation) instructions, which is _tower-nf_ by default. 1. Specify the **Head service account** created in the [cluster preparation](#cluster-preparation) instructions, which is _tower-launcher-sa_ by default. 1. Specify the **Storage claim** created in the [cluster preparation](#cluster-preparation) instructions, which serves as a scratch filesystem for Nextflow pipelines. The storage claim is called _tower-scratch_ in each of the provided examples. 1. Apply [**Resource labels**](../resource-labels/overview) to the cloud resources consumed by this compute environment. Workspace default resource labels are prefilled. 1. Expand **Staging options** to include: - Optional [pre- or post-run Bash scripts](../launch/advanced#pre-and-post-run-scripts) that execute before or after the Nextflow pipeline execution in your environment. - Global Nextflow configuration settings for all pipeline runs launched with this compute environment. Values defined here are pre-filled in the **Nextflow config file** field in the pipeline launch form. These values can be overridden during pipeline launch. :::info Configuration settings in this field override the same values in the pipeline repository `nextflow.config` file. See [Nextflow config file](../launch/advanced#nextflow-config-file) for more information on configuration priority. ::: 1. Under **Environment variables**, add each variable with a **Name**, **Value**, and **Target Environment**: - **Head job**: Adds the variable to the Nextflow head job container, which evaluates `nextflow.config` and submits tasks to the compute backend. Use this target for variables that Nextflow or its plugins read, such as `NXF_OPTS`, `NXF_JVM_ARGS`, `NXF_PLUGINS_DEFAULT`, or proxy settings the head node uses to reach external services. - **Compute job**: Adds the variable to the worker containers that run individual pipeline tasks. Use this target for variables your pipeline tools read, such as `OPENAI_API_KEY` for a process that calls the OpenAI API, registry credentials needed inside the task container, or tool-specific settings like `JAVA_HOME`. - **Head and Compute jobs**: Adds the variable to both the head job and the compute jobs. Use this target for values needed in both places, such as an HTTP proxy used by both Nextflow and task tools, or a credential needed in both the head job and individual compute tasks. :::note For sensitive values such as API keys and tokens, use [pipeline secrets](../secrets/overview) instead of custom environment variables. Custom environment variables are stored in the compute environment configuration and cannot be edited after creation. To rotate a value, recreate the compute environment. ::: 1. Configure any advanced options described below, as needed. 1. Select **Create** to finalize the compute environment setup. See [Launch pipelines](../launch/launchpad) to start executing workflows in your Kubernetes compute environment. ### Advanced options Seqera Platform compute environments for Kubernetes include advanced options for storage and work directory paths, resource allocation, and pod customization. **Seqera Kubernetes advanced options** - The **Storage mount path** is the file system path where the Storage claim is mounted (default: `/scratch`). - The **Work directory** is the file system path used as a working directory by Nextflow pipelines. It must be the storage mount path (default) or a subdirectory of it. - The **Compute service account** is the service account used by Nextflow to submit tasks (default: the `default` account in the given namespace). - The **Pod cleanup policy** determines when Seqera deletes the run's Nextflow head pod. **On success** (the default) deletes it when the run succeeds, **Always** deletes it regardless of outcome, and **Never** keeps it. This policy does not delete task pods. See [Kubernetes pods remain after a run ends](../troubleshooting_and_faqs/nextflow#kubernetes-pods-remain-after-a-run-ends). - Use **Custom head pod specs** to provide custom options for the Nextflow workflow pod (`nodeSelector`, `affinity`, etc). For example: ```yaml spec: nodeSelector: disktype: ssd ``` - Use **Custom service pod specs** to provide custom options for the compute environment pod. See above for an example. - Use **Head job CPUs** and **Head job memory** to specify the hardware resources allocated to the Nextflow workflow pod. --- ## Compute environment overview Seqera Platform **compute environments** define the execution platform where a pipeline will run. Compute environments enable users to launch pipelines on a growing number of **cloud** and **on-premises** platforms. Each compute environment must be configured to enable Seqera to submit tasks. See the individual compute environment pages below for platform-specific configuration steps. ## Platforms - [AWS Batch](./aws-batch) - [AWS Cloud](./aws-cloud) - [Azure Batch](./azure-batch) - [Google Batch](./google-cloud-batch) - [Google Cloud](./google-cloud) - [Grid Engine](./hpc) - [Altair PBS Pro](./hpc) - [IBM LSF](./hpc) - [Moab](./hpc) - [Slurm](./hpc) - [Kubernetes](./k8s) - [Amazon EKS](./eks) - [Google Kubernetes Engine](./gke) :::note Compute Environments now support descriptions. Enter a description during creation to provide context and information. To update a description, select **Edit** from the menu of the relevant compute environment. You can update descriptions at any time (e.g., to reflect a status change), up to a 1000-character limit. You can also add descriptions to existing compute environments that don't have one. ::: ## Select default compute environment If you have more than one compute environment, you can select a workspace primary compute environment to be used as the default when launching pipelines in that workspace. In a workspace, select **Compute Environments**. Then select **Make primary** from the options menu next to the compute environment you wish to use as default. ## Rename compute environment You can edit the names of compute environments in private and organization workspaces. Select **Rename** from the options menu next to the compute environment you wish to edit. Select **Update** on the edit page to save your changes after you have updated the compute environment name. ## Disable compute environment Users with **Admin** or **Owner** [workspace permissions](../orgs-and-teams/roles#workspace-participant-roles) can disable and enable compute environments. When you disable a compute environment: - Actions that use this compute environment will fail to run. **Update actions to use a new compute environment**. - New pipelines and Studio sessions will not run on the disabled compute environment. **Update pipelines and Studios to use a new compute environment**. - **Running pipelines and Studio sessions are not terminated**. Ongoing runs and Studio sessions will finish gracefully. - If the compute environment was set as primary, it will be unset. Until you select a new primary compute environment, new runs will default to the next available compute environment. To disable a compute environment, select **Disable** from the options menu next to the compute environment in your workspace **Compute Environments** page. To re-enable a disabled compute environment, select **Enable** from the options menu. Enabled compute environments can run new pipelines and Studio sessions. ## Export compute environment You can export a compute environment's configuration as a JSON file for troubleshooting, audits, or as a reference when recreating it. :::note The exported JSON is for reference only. Re-importing it through the Seqera Platform UI is not supported. ::: Any user with the Maintain, Launch, or View role on the workspace can export. The compute environment detail page, or the form page for a specific compute environment. **What's included**: - Name, platform, region, and work directory - Forge or manual configuration block - Fusion and Wave settings - Environment variables - Pre- and post-run scripts - Labels - A reference to the credential used (the credential itself is excluded) **What's not included**: - Credentials and secrets ## Disable compute environment Users with **Admin** or **Owner** [workspace permissions](../orgs-and-teams/roles#workspace-participant-roles) can disable and enable compute environments. When you disable a compute environment: - Actions that use this compute environment will fail to run. **Update actions to use a new compute environment**. - New pipelines and Studio sessions will not run on the disabled compute environment. **Update pipelines and Studios to use a new compute environment**. - **Running pipelines and Studio sessions are not terminated**. Ongoing runs and Studio sessions will finish gracefully. - If the compute environment was set as primary, it will be unset. Until you select a new primary compute environment, new runs will default to the next available compute environment. To disable a compute environment, select **Disable** from the options menu next to the compute environment in your workspace **Compute Environments** page. To re-enable a disabled compute environment, select **Enable** from the options menu. Enabled compute environments can run new pipelines and Studio sessions. ## Delete compute environment Compute environments can be deleted when they are no longer required. You must delete the compute environment before deleting its associated credentials. If the credentials are deleted first, the compute environment deletion will fail with an error. If this happens, the entry needs to be manually deleted from the database to fully remove it. ## GPU usage The process for provisioning GPU instances in your compute environment differs for each cloud provider. ### AWS Batch The AWS Batch compute environment creation form in Seqera includes an **Enable GPUs** option. This enables you to run GPU-dependent workflows in the compute environment. Some important considerations: - Seqera only supports NVIDIA GPUs. Select instances with NVIDIA GPUs for your GPU-dependent processes. - The **Enable GPUs** setting causes Batch Forge to specify the most current [AWS-recommended GPU-optimized ECS AMI](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI.html) as the EC2 fleet AMI when creating the compute environment. This setting can be overridden by **AMI ID** in the advanced options. - The **Enable GPUs** setting alone does not deploy GPU instances in your compute environment. You must still specify GPU-enabled instance types in the **Advanced options > Instance types** field. - Your Nextflow script must include [accelerator directives](https://docs.seqera.io/nextflow/process.html?highlight=accelerator#accelerator) to use the provisioned GPUs. - The NVIDIA Container Runtime uses [environment variables](https://github.com/NVIDIA/nvidia-container-runtime#environment-variables-oci-spec) in container images to specify a GPU accelerated container. These variables should be included in the [`containerOptions`](https://docs.seqera.io/nextflow/process#process-containeroptions) directive for each GPU-dependent process in your Nextflow script. The `containerOptions` directive can be set inline in your process definition or via configuration. For example, to add the directive to a process named `UseGPU` via configuration: ```groovy process { withName: UseGPU { containerOptions '-e NVIDIA_DRIVER_CAPABILITIES=compute,utility -e NVIDIA_VISIBLE_DEVICES=all' } } ``` - GPU-accelerated containers (such as NVIDIA Parabricks) bundle a specific CUDA runtime. The compute environment's AMI must include an NVIDIA driver compatible with the container's CUDA runtime. When **Enable GPUs** is set, Batch Forge selects the current AWS-recommended GPU-optimized ECS AMI. If you override **AMI ID** under **Advanced options**, confirm that the custom AMI's driver satisfies the container's CUDA version. See the [NVIDIA CUDA compatibility matrix](https://docs.nvidia.com/deploy/cuda-compatibility/) for supported driver versions. For GPU driver and CUDA compatibility errors, see [AWS troubleshooting](../troubleshooting_and_faqs/aws_troubleshooting#gpus). ### GPU metrics :::note Detailed GPU metrics are only available for tasks that run with Fusion version 2.5.10 onwards and using Nextflow version 26.03.3-edge onwards ::: When [Fusion](https://docs.seqera.io/fusion) is enabled, Seqera Platform automatically collects GPU metrics for tasks that run on NVIDIA GPU instances. No additional configuration is required beyond enabling Fusion and provisioning GPU instances in your compute environment. The following metrics are collected per task: - **GPU type**: The GPU model (e.g., NVIDIA A10G, A100). - **Driver version**: The NVIDIA driver version in use. - **GPU utilization %**: The percentage of GPU compute capacity used. - **GPU memory peak**: The maximum GPU memory used during execution. - **GPU memory average**: The average GPU memory used during execution. For tasks that use multiple GPUs, metrics are aggregated (average or peak across all GPUs assigned to the task) and displayed as a single combined value per task. #### Where GPU metrics appear - **Task detail view**: Select a GPU task in the task table to view GPU type, driver version, utilization, and memory metrics alongside existing CPU metrics. ![GPU metrics in task detail view](./_images/gpu-metrics-task.png) - **Metrics tab**: A dedicated **GPU** section displays box-and-whisker plots grouped by task name, with tabs for **GPU Utilization %**, **Memory Peak**, and **Memory Average**. This section appears only when the workflow includes tasks with GPU data. ![GPU utilization](./_images/gpu-metrics-utilization.png) ![GPU memory peak](./_images/gpu-metrics-memory-peak.png) ![GPU memory average](./_images/gpu-metrics-memory-average.png) - **Platform API**: GPU metrics are included in [task](https://docs.seqera.io/platform-api/describe-workflow-task) and [workflow](https://docs.seqera.io/platform-api/list-workflow-tasks) API responses for programmatic access. :::note GPU metrics are only available for tasks that run with Fusion enabled on NVIDIA GPU instances. Non-GPU tasks do not display a GPU metrics section. For tasks that fail mid-execution, partial metrics collected up to the point of failure are shown. ::: --- ## Compute environment pre-flight checks Pre-flight checks confirm that a compute environment is usable before you launch a pipeline. They run when you create or update a compute environment, on a recurring background schedule, and again at launch time. Problems surface before submission rather than mid-run. Pre-flight checks only flag conditions that would block a launch. Seqera Platform Enterprise enables pre-flight checks and credential validation by default. An administrator can disable either with the environment variables described in [Feature flags](#feature-flags). ## Checked conditions Pre-flight checks confirm the following conditions. Verify them before you create a compute environment: **Credentials** - The access keys, service account key, or managed identity are valid and have not been rotated or revoked. - The IAM role or service account has the permissions the cloud provider requires. See the relevant compute environment page for the minimum required policy. **Wave** (if enabled) - The Wave service is running and reachable from Seqera Platform. **Tower Agent** (high-performance computing (HPC) and grid compute environments only) - Tower Agent is reachable from Platform. See [Tower Agent](../supported_software/agent/overview) for installation and startup instructions. ## Feature flags Two flags control pre-flight checks and credential validation. Both default to `true`. A new installation runs every check described on this page with no extra configuration. To disable pre-flight checks, set both flags to `false` in `tower.yml`. | Environment variable | `tower.yml` key | What it does | Default | |---|---|---|---| | `TOWER_CREDENTIALS_VALIDATION_ENABLED` | `tower.credentials.validation.enabled` | Validates credentials against the cloud provider when you create or update them, stores the result on the credential record, and shows the status in the UI. When `false`, Platform skips this validation, hides the status block in the UI, and `/credentials/{id}/validate` returns a result without saving it. | `true` | | `TOWER_PREFLIGHT_CHECK_ENABLED` | `tower.preflight.check.enabled` | Runs the credential and compute environment validation crons, rejects launches against `INVALID` credentials or compute environments with a `400 Bad Request`, and hides `INVALID` credentials in the compute environment creation picker. When `false`, the crons do not run, the launch API ignores `INVALID` status, and the picker shows every credential. | `true` | :::note Platform reads both flags once at process start. Restart the `backend` and `cron` containers after changing either value. ::: :::warning[Disable both flags together] If you set `TOWER_CREDENTIALS_VALIDATION_ENABLED` to `false` but leave `TOWER_PREFLIGHT_CHECK_ENABLED` at `true`, the background cron still marks failing credentials `INVALID` and Platform still rejects launches against them. The **Credentials** page hides the `INVALID` status and the **Validate** action, and updating the credential does not re-validate it. ::: ## Validation process Validation runs at four points: ### Compute environment creation and update checks This check runs when you create a compute environment, and when you update one to use different credentials. Platform reads the stored status of the selected credential. If the credential has been deleted or is `INVALID`, Platform rejects the request with a `400 Bad Request` that names the credential and the recovery step. The check runs before any provider validation. This check does not apply to compute environments that use a managed identity, because no credential is attached. On update, Platform checks only the newly selected credential. Platform does not block other edits, such as a name or description change. To restore a compute environment whose credential has failed, replace the `INVALID` or deleted credential with a working one. :::note This is the only check that reads credential status when `TOWER_PREFLIGHT_CHECK_ENABLED` is `false`. In that state, Platform rejects a new compute environment that uses an `INVALID` credential, but does not block launches against a compute environment that already uses one. The check only fires where credential validation has run at some point, because nothing else writes the `INVALID` status. On an installation that has always run with both flags set to `false`, every credential is `AVAILABLE` and this check never fires. ::: ### Credential validation This check runs on a recurring schedule. For each AWS, Google Cloud, and Azure credential in scope, Platform calls the provider API to confirm that the credential is still accepted. For AWS role-based credentials and Google Cloud Workload Identity Federation, the check confirms the credential is well-formed. It cannot fully verify the underlying role or identity provider trust configuration. When a credential fails this check, Platform marks it `INVALID` and records the provider error on the credential record. The error appears in the launch-time error message when a pipeline is blocked, but not in the compute environment banner. To see the provider error, check the credential record. A transient probe failure, such as a network interruption or provider throttling, does not mark the credential `INVALID`. Platform retries the credential with exponential backoff. Some failures show that the credential can never validate, such as an Azure storage or Batch account hostname that no longer resolves. After 10 consecutive failures of this kind, Platform marks the credential `INVALID`. See [Credential validation cron](#credential-validation-cron) to tune the retry ceiling and the escalation threshold. ### Compute environment validation This check runs on a recurring schedule. Platform reads the status of the associated credential. If the credential is `INVALID`, Platform marks the compute environment `INVALID` immediately. An `INVALID` compute environment displays a banner with the error message. For an `AVAILABLE` compute environment, Platform refreshes the `lastValidated` timestamp. :::note This check covers AWS Batch, AWS Cloud, Azure Batch, Azure Cloud, Google Cloud Batch, and Google Cloud compute environments. ::: ### Pipeline launch-time checks These checks run when a user submits a pipeline launch. If any check fails, Platform blocks the launch and reports every failure in one error. | Check | What it does | |---|---| | Compute environment status | Reads the last recorded status from the database. Blocks the launch if the compute environment is `INVALID`. | | Credential status | Reads the last recorded status from the database. Blocks the launch if the credential associated with the compute environment is `INVALID`. | | Wave connectivity | For compute environments with Wave enabled, confirms that the Wave service connection is active. | | Tower Agent | For HPC compute environments, confirms that a Tower Agent is online for the environment. | ## Validate a credential manually After you rotate the keys or fix the underlying issue on an `INVALID` credential, trigger an immediate re-validation: 1. Go to **Credentials** in your workspace. 2. Find the credential, open its **⋮** menu, and select **Validate**. Platform makes a live call to the cloud provider and updates the credential status immediately. If the check passes, the credential returns to `AVAILABLE`. The **Validate** action is available only while the credential is `INVALID`. Compute environments marked `INVALID` because of this credential do not recover automatically. Use **Validate** on each affected compute environment after restoring the credential. ## Validate a compute environment manually After you fix the underlying issue on an `INVALID` compute environment, trigger an immediate re-validation without waiting for the next background sweep: 1. Go to **Compute environments** in your workspace. 2. Find the compute environment, open its **⋮** menu, and select **Validate**. Platform runs pre-flight checks and updates the compute environment status immediately. If all checks pass, the compute environment returns to `AVAILABLE`. :::warning[Validate the credential before the compute environment] If both the credential and its associated compute environment are `INVALID`, restore the credential to `AVAILABLE` first. If the credential is still `INVALID`, the compute environment remains `INVALID`. ::: ## Validation settings The defaults work for most deployments. Adjust these settings only if you have specific rate-limit or scheduling requirements. The cron settings have no effect when `TOWER_PREFLIGHT_CHECK_ENABLED` is `false`. ### Credential validation cron | Environment variable | `tower.yml` key | Description | Default | |---|---|---|---| | `TOWER_CRON_CREDENTIALS_VALIDATION_INTERVAL` | `tower.cron.credentials-validation.interval` | Per-credential re-validation cadence. After each successful probe, Platform reschedules the credential at `now + interval ± 10%` jitter. Because the launch-time check independently blocks launches on revoked credentials, you can relax this cadence. | `12h` | | `TOWER_CRON_CREDENTIALS_VALIDATION_TICK_RATE` | `tower.cron.credentials-validation.tick-rate` | How often the evaluator polls the Redis schedule store for due credentials. Distinct from the per-credential interval. | `60s` | | `TOWER_CRON_CREDENTIALS_VALIDATION_DELAY` | `tower.cron.credentials-validation.delay` | Initial delay before the first evaluator tick after process start. Randomized ±50% to spread cold-start load across replicas. | `20s` | | `TOWER_CRON_CREDENTIALS_VALIDATION_BATCH_SIZE` | `tower.cron.credentials-validation.batch-size` | Maximum credential IDs drained from the Redis schedule store per evaluator tick. | `100` | | `TOWER_CRON_CREDENTIALS_VALIDATION_CONCURRENCY` | `tower.cron.credentials-validation.concurrency` | Global cap on concurrent cloud probes across all evaluator pumps. Lower it when many credentials in one workspace share a single cloud account, to avoid provider rate limits such as AWS STS `TooManyRequests`. | `10` | | `TOWER_CRON_CREDENTIALS_VALIDATION_PROBE_DELAY` | `tower.cron.credentials-validation.probe-delay` | Optional pause between probes within a single pump. Set a non-zero value, for example `200ms`, when many credentials share a cloud account and a cold-start burst would exceed provider rate limits. | `0ms` (no pacing) | | `TOWER_CRON_CREDENTIALS_VALIDATION_TRANSIENT_RETRY_INTERVAL` | `tower.cron.credentials-validation.transient-retry-interval` | Cadence for re-enqueuing a credential after a transient probe failure, such as a network interruption, a provider 5xx, or an unexpected SDK exception. Without it, Platform skips a failed credential until the next process restart. | `5m` | | `TOWER_CRON_CREDENTIALS_VALIDATION_TRANSIENT_RETRY_MAX_INTERVAL` | `tower.cron.credentials-validation.transient-retry-max-interval` | Ceiling on the transient retry delay. The delay doubles after each consecutive transient failure, starting from `TOWER_CRON_CREDENTIALS_VALIDATION_TRANSIENT_RETRY_INTERVAL` (5m, 10m, 20m, 40m, up to this ceiling). As a result, Platform probes a persistently failing credential less often over time. Must be greater than or equal to the transient retry interval, or the process fails at startup. | `24h` | | `TOWER_CRON_CREDENTIALS_VALIDATION_UNVERIFIABLE_MAX_ATTEMPTS` | `tower.cron.credentials-validation.unverifiable-max-attempts` | Number of consecutive unverifiable probe failures before Platform marks the credential `INVALID`. Only a DNS resolution failure on a hostname derived from the credential itself counts as unverifiable. This applies to Azure storage and Batch account hostnames. Platform never escalates fixed provider endpoints, such as `sts.amazonaws.com`. Must be `1` or greater, or the process fails at startup. To disable escalation and keep backoff only, set a very high value and also lower `TOWER_CRON_CREDENTIALS_VALIDATION_TRANSIENT_RETRY_MAX_INTERVAL`, for example to `21h`. With the default `24h` ceiling, values above `10` fail the startup check described later. | `10` (about 1.8 days) | The consecutive-failure counters behind these two settings expire 24 hours after the last failed probe. At startup, Platform verifies that the configured combination cannot delay the final escalation attempt past that window. If it can, the process fails to start with an error that names both settings. Lower one of the two values to resolve it. ### Compute environment validation cron | Environment variable | `tower.yml` key | Description | Default | |---|---|---|---| | `TOWER_CRON_COMPUTE_ENV_VALIDATION_INTERVAL` | `tower.cron.compute-env-validation.interval` | Compute environment re-validation cadence. A compute environment is due when its `lastValidated` timestamp is null or older than `now - interval`. | `12h` | | `TOWER_CRON_COMPUTE_ENV_VALIDATION_TICK_RATE` | `tower.cron.compute-env-validation.tick-rate` | How often the evaluator sweeps the database for due compute environments. Distinct from the per-compute-environment interval. | `60s` | | `TOWER_CRON_COMPUTE_ENV_VALIDATION_DELAY` | `tower.cron.compute-env-validation.delay` | Initial delay before the first evaluator tick after process start. Randomized ±50% to spread cold-start load across replicas. | `20s` | | `TOWER_CRON_COMPUTE_ENV_VALIDATION_BATCH_SIZE` | `tower.cron.compute-env-validation.batch-size` | Maximum compute environment IDs swept from the database per evaluator tick. | `100` | ### Credential auto-validation timeout This setting has no effect when `TOWER_CREDENTIALS_VALIDATION_ENABLED` is `false`. | Environment variable | `tower.yml` key | Description | Default | |---|---|---|---| | `TOWER_CREDENTIALS_AUTO_VALIDATION_TIMEOUT_SEC` | `tower.credentials.autoValidationTimeoutSec` | Timeout in seconds for the cloud provider probe that runs when you create or update a credential. Platform treats timeout expiry as a transient failure and leaves the stored status untouched. | `10` | ## Error reference For pre-flight check error messages, causes, and resolutions, see [Pre-flight checks troubleshooting](../troubleshooting_and_faqs/preflight_checks_troubleshooting). --- ## Tower Agent credentials [Tower Agent](../supported_software/agent/overview) enables Seqera Platform to launch pipelines on HPC clusters that do not allow direct access through an SSH client. Tower Agent authenticates a secure connection with Seqera using a Tower Agent credential. ## Tower Agent sharing You can share a single Tower Agent instance with all members of a workspace. Create a Tower Agent credential, with **Shared agent** enabled, in the relevant workspace. All workspace members can then use this credential (Connection ID + Seqera access token) to use the same Tower Agent instance. ## Create a Tower Agent credential 1. Add your credentials to your organization or personal workspace: - From an organization workspace: Go to **Credentials > Add Credentials**. - From your personal workspace: From the user menu, go to **Your credentials > Add credentials**. 2. Complete the following fields: - **Name**: Specify a unique name for the credentials using alphanumeric characters, dashes, or underscores. For example, `my-agent-creds`. - **Provider**: Select **Tower Agent**. - **Agent connection ID**: The connection ID used to run your Tower Agent instance. Must match the connection ID used when running the Agent (see **Usage** below). - **Shared agent**: Enables Tower Agent sharing for all workspace members. - **Usage**: Populates a code snippet for Tower Agent download with your connection ID. Replace `` with your [Seqera access token](https://docs.seqera.io/platform-api/create-token). 3. After you've completed all the form fields, select **Add**. The new credential is now listed under the **Credentials** tab. --- ## AWS ECR credentials From version 22.3, Seqera Platform supports the configuration of credentials for the Nextflow Wave container service to authenticate to private and public container registries. For more information on Wave containers, see [the Nextflow documentation](https://docs.seqera.io/nextflow/wave). :::note Container registry credentials are only used by the Wave container service. Add `wave { enabled=true }` to the **Nextflow config** field on the launch page, or to your `nextflow.config` file, for your pipeline execution to use Wave containers. ::: ## AWS ECR Private Registry Wave requires programmatic access to your private Elastic Container Registry (ECR) via [long-term access keys](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#create-long-term-access-keys). Create a user with registry read permissions (e.g., a subset of the AWS-managed `AmazonEC2ContainerRegistryReadOnly` policy) for this purpose. **Create an IAM user with AWS ECR access** 1. Open the [IAM console](https://console.aws.amazon.com/iam/). 2. Select **Users** from the navigation pane. 3. Select the name of the user whose keys you want to manage, then select the **Security credentials** tab. We recommend creating an IAM user specifically for Wave authentication instead of using existing credentials with broader permissions. 4. In the **Access keys** section, select **Create access key**. Each IAM user can have only two access keys at a time, so if the Create option is deactivated, delete an existing access key first. 5. On the **Access key best practices & alternatives** page, select **Other** and then **Next**. 6. On the **Retrieve access key** page, you can either **Show** the user's secret access key details, or store them by selecting **Download .csv file**. 7. The newly created access key pair is active by default and can be stored as a container registry credential in Seqera. :::note Your credential must be stored in Seqera as a **container registry** credential, even if the same access keys already exist as a workspace credential. ::: ## Add private ECR credentials to Seqera 1. Add your credentials to your organization or personal workspace: - From an organization workspace: Go to **Credentials > Add Credentials**. - From your personal workspace: From the user menu, go to **Your credentials > Add credentials**. 2. Complete the following fields: - **Name**: Specify a unique name for the credentials using alphanumeric characters, dashes, or underscores. For example, `my-registry-creds`. - **Provider**: Select **Container registry**. - **User name**: Specify your IAM user access key ID. For example, `AKIAIOSFODNN7EXAMPLE`. - **Password**: Specify your IAM user secret access key. - **Registry server**: Specify your private ECR registry URL. For example, `.dkr.ecr..amazonaws.com`. 3. After you've completed all the form fields, select **Add**. The new credential is now listed under the **Credentials** tab. ## AWS ECR Public Registry Amazon ECR Public Gallery (`public.ecr.aws`) hosts publicly accessible container images. While images can be pulled without authentication, AWS applies rate limits to unauthenticated pulls. Authenticating with AWS credentials removes these rate limits and is required when running pipelines at scale. ### Required IAM permissions The IAM user needs the following permissions to authenticate to ECR Public: - `ecr-public:GetAuthorizationToken` - `ecr-public:BatchCheckLayerAvailability` - `ecr-public:GetRepositoryPolicy` - `ecr-public:DescribeRepositories` - `ecr-public:DescribeImages` - `ecr-public:DescribeImageTags` - `sts:GetServiceBearerToken` Attach the AWS managed policy `AmazonElasticContainerRegistryPublicReadOnly` and add the `sts:GetServiceBearerToken` permission. This permission is not included in the managed policy and must be granted separately, or ECR Public authentication will fail. ### Add ECR Public credentials to Seqera 1. Add your credentials to your organization or personal workspace: - From an organization workspace: Go to **Credentials > Add Credentials**. - From your personal workspace: From the user menu, go to **Your credentials > Add credentials**. 2. Complete the following fields: - **Name**: Specify a unique name for the credentials using alphanumeric characters, dashes, or underscores. For example, `ecr-public-creds`. - **Provider**: Select **Container registry**. - **User name**: Specify your IAM user access key ID. For example, `AKIAIOSFODNN7EXAMPLE`. - **Password**: Specify your IAM user secret access key. - **Registry server**: Enter `public.ecr.aws`. 3. After you've completed all the form fields, select **Add**. Wave matches the `public.ecr.aws` hostname to these credentials and authenticates ECR Public pulls on your behalf. --- ## Azure container registry credentials From version 22.3, Seqera Platform supports the configuration of credentials for the Nextflow Wave container service to authenticate to private and public container registries. For more information on Wave containers, see the [Nextflow documentation](https://docs.seqera.io/nextflow/wave). :::note Container registry credentials are only used by the Wave container service. Add `wave { enabled=true }` to the **Nextflow config** field on the launch page, or to your `nextflow.config` file, for your pipeline execution to use Wave containers. ::: ## Azure container registry access Azure container registry makes use of Azure RBAC (Role-Based Access Control) to grant users access. For more information, see [Azure container registry roles and permissions](https://learn.microsoft.com/en-us/azure/container-registry/container-registry-roles). You must use Azure credentials with long-term registry read (**content/read**) access to authenticate Seqera to your registry. We recommend a [token with repository-scoped permissions](https://learn.microsoft.com/en-us/azure/container-registry/container-registry-repository-scoped-permissions) that's used only by Seqera. **Create an access token with Azure container registry access** 1. In the Azure portal, navigate to your container registry. 2. Under **Repository permissions**, select **Tokens > +Add**. 3. Enter a token name. 4. Under **Scope map**, select **Create new**. 5. In the **Create scope map** section, enter a name and description for the new scope map. 6. Select your **Repository** from the drop-down. 7. Select **content/read** from the **Permissions** drop-down, then select **Add** to create the scope map. 8. In the **Create token** section, ensure the **Status** is **Enabled** (default), then select **Create**. 9. Return to **Repository permissions > Tokens** for your registry, then select the token you just created. 10. On the token details page, select **password1** or **password2**. 11. In the password details section, uncheck the **Set expiration date?** checkbox, then select **Generate**. 12. Copy and save the generated password (this is only displayed once). ## Add credentials to Seqera 1. Add your credentials to your organization or personal workspace: - From an organization workspace: Go to **Credentials > Add Credentials**. - From your personal workspace: From the user menu, go to **Your credentials > Add credentials**. 2. Complete the following fields: - **Name**: Specify a unique name for the credentials using alphanumeric characters, dashes, or underscores. For example, `my-registry-creds`. - **Provider**: Select **Container registry**. - **User name**: Specify your registry token name. For example, `my-registry-token`. - **Password**: Your registry token password. For example, `my-registry-token`. - **Registry server**: Specify the container registry server name. You can obtain this from the Azure portal: **Settings > Access keys > Login server**. For example, `myregistry.azurecr.io`. 3. After you've completed all the form fields, select **Add**. The new credential is now listed under the **Credentials** tab. --- ## Data repositories Data Explorer requires programmatic access via valid credentials to browse and interact with remotely hosted private data repositories. To automatically connect to one or more data repositories, create a new credential that includes **Name** and **Provider**. Specific data repositories require additional information to connect. ## AWS Simple Storage Service (S3) object storage Add an **Access key**, and **Secret key**. You can optionally provide an IAM role for temporary access - this must be a fully qualified AWS role ARN. S3 object storage buckets are prefixed with an AWS icon and `s3://` in Data Explorer. :::note Seqera Compute uses AWS S3 object storage, and are prefixed with a Seqera icon and the `s3://` namespace in Data Explorer. ::: ## Azure Blob Storage Select between different credential types: a **Shared key**, **Entra**, or **Cloud**. - **Shared key:** Access your Azure accounts directly using primary or secondary access keys. - **Entra:** Authenticate via an Azure Entra service principal for enhanced security and identity management. - **Cloud:** Authenticate via an Azure Entra service principal for Azure Cloud. :::info Select Entra for modern, identity-based access control. Select Cloud for Entra identity-based access control with Cloud specializations. Select Shared key for full, direct account access. ::: Add a **Batch account name**, **Batch account key**, **Blob Storage account name**, and **Blob Storage account key**. Azure Blob Storage are prefixed with an Azure icon and `az://` in Data Explorer. ## GCP object storage Add the contents of the **Service account key** JSON file. GCP object storage buckets are prefixed with a GCP icon and `gs://` in Data Explorer. ## S3-compatible storage This includes cloud-provider and on-premise based storage solutions with an S3-compatible API. Examples include [Cloudflare R2][cloudflare], [MinIO][minio], and [Oracle Cloud Infrastructure][oci]. Add an **Access key**, **Secret key**, **Server base URL**, and optionally select path-style URL access. Refer to your S3-compatible storage provider documentation to determine if path-style URL access is applicable. :::info OCI has specific object-storage endpoints that are [S3-compatible][oci-s3-compatible], and include `.compat.` in the server base URL. These are in the form `https://.compat.objectstorage..oci.customer-oci.com`. ::: S3-compatible storage are prefixed with a S3-compatible storage icon and `s3://` in Data Explorer. {/* Links */} [cloudflare]: https://www.cloudflare.com/developer-platform/products/r2/ [minio]: https://min.io [oci]: https://www.oracle.com/cloud/ [oci-s3-compatible]: https://docs.oracle.com/en-us/iaas/api/#/en/s3objectstorage --- ## Docker Hub credentials From version 22.3, Seqera Platform supports the configuration of credentials for the Nextflow Wave container service to authenticate to private and public container registries. For more information on Wave containers, see the [Nextflow documentation](https://docs.seqera.io/nextflow/wave). :::note Container registry credentials are only used by the Wave container service. Add `wave { enabled=true }` to the **Nextflow config** field on the launch page, or to your `nextflow.config` file, for your pipeline execution to use Wave containers. ::: ## Docker Hub registry access You must use Docker Hub credentials with **Read-only** access to authenticate Seqera to your registry. Docker Hub uses personal access tokens (PATs) for authentication. We don't currently support Docker Hub authentication with 2FA (two-factor authentication). **Create a Docker Hub PAT** 1. Log in to [Docker Hub](https://hub.docker.com/). 2. Select your username in the top right corner and select **Account Settings**. 3. Select **Security > New Access Token**. 4. Enter a token description and select **Read-only** from the Access permissions drop-down, then select **Generate**. 5. Copy and save the generated access token (this is only displayed once). ## Add credentials to Seqera 1. Add your credentials to your organization or personal workspace: - From an organization workspace: Go to **Credentials > Add Credentials**. - From your personal workspace: From the user menu, go to **Your credentials > Add credentials**. 2. Complete the following fields: - **Name**: Specify a unique name for the credentials using alphanumeric characters, dashes, or underscores. For example, `my-registry-creds`. - **Provider**: Select **Container registry**. - **User name**: Specify your Docker username. For example, `user1`. - **Password**: Specify your personal access token (PAT). For example, `1fcd02dc-...215bc3f3`. - **Registry server**: Specify the container registry hostname, excluding the protocol. For example, `docker.io`. 3. After you've completed all the form fields, select **Add**. The new credential is now listed under the **Credentials** tab. --- ## Gitea container registry credentials From version 22.3, Seqera Platform supports the configuration of credentials for the Nextflow Wave container service to authenticate to private and public container registries. For more information on Wave containers, see the [Nextflow documentation](https://docs.seqera.io/nextflow/wave). Gitea container registries support [authentication][gitea-auth] using a personal access token. Use your personal access token as your password when you create your Gitea container registry credentials in Seqera. :::note Container registry credentials are only used by the Wave container service. Add `wave { enabled=true }` to the **Nextflow config** field on the launch page, or to your `nextflow.config` file, for your pipeline execution to use Wave containers. ::: ## Create a personal access token (PAT) You must create a PAT to access your Gitea container registry from Wave. For more information, see [Create a personal access token][gitea-create]. ## Add credentials to Seqera 1. Add your credentials to your organization or personal workspace: - From an organization workspace: Go to **Credentials > Add Credentials**. - From your personal workspace: From the user menu, go to **Your credentials > Add credentials**. 2. Complete the following fields: - **Name**: Specify a unique name for the credentials using alphanumeric characters, dashes, or underscores. For example, `my-registry-creds`. - **Provider**: Select **Container registry**. - **User name**: Specify your Gitea username. For example, `gitlab_user1`. - **Password**: Specify your Gitea personal access token (PAT). For example, `1fcd02dc-...215bc3f3`. - **Registry server**: Specify your Gitea container registry URL. For example, `gitea.example.com`. 3. After you've completed all the form fields, select **Add**. The new credential is now listed under the **Credentials** tab. [gitea-auth]: https://docs.gitea.com/usage/packages/container#login-to-the-container-registry [gitea-create]: https://docs.gitea.com/development/api-usage#authentication --- ## GitHub container registry credentials From version 22.3, Seqera Platform supports the configuration of credentials for the Nextflow Wave container service to authenticate to private and public container registries. For more information on Wave containers, see the [Nextflow documentation](https://docs.seqera.io/nextflow/wave). GitHub Packages only supports [authentication][github-pat] using a personal access token (classic). Use your personal access token as your password when you create your GitHub container registry credentials in Seqera. :::note Container registry credentials are only used by the Wave container service. Add `wave { enabled=true }` to the **Nextflow config** field on the launch page, or to your `nextflow.config` file, for your pipeline execution to use Wave containers. ::: ## Create a personal access token (PAT) You must create a PAT to access your GitHub container registry from Wave. For more information, see [Create a personal access token][github-create]. ## Add credentials to Seqera 1. Add your credentials to your organization or personal workspace: - From an organization workspace: Go to **Credentials > Add Credentials**. - From your personal workspace: From the user menu, go to **Your credentials > Add credentials**. 2. Complete the following fields: - **Name**: Specify a unique name for the credentials using alphanumeric characters, dashes, or underscores. For example, `my-registry-creds`. - **Provider**: Select **Container registry**. - **User name**: Specify your GitHub username. For example, `github_user1`. - **Password**: Specify your personal access token (PAT) classic. For example, `1fcd02dc-...215bc3f3`. - **Registry server**: Specify your GitHub container registry URL. For example, `ghcr.io`. 3. After you've completed all the form fields, select **Add**. The new credential is now listed under the **Credentials** tab. [github-pat]: https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-with-a-personal-access-token-classic [github-create]: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token-classic --- ## GitLab container registry credentials From version 22.3, Seqera Platform supports the configuration of credentials for the Nextflow Wave container service to authenticate to private and public container registries. For more information on Wave containers, see the [Nextflow documentation](https://docs.seqera.io/nextflow/wave). If your organization enabled two-factor authentication (2FA) for your GitLab organization or project, you must use your [personal access token][gitlab-pat] as your password when you create your [GitLab container registry credentials][gitlab-cr]. :::note Container registry credentials are only used by the Wave container service. Add `wave { enabled=true }` to the **Nextflow config** field on the launch page, or to your `nextflow.config` file, for your pipeline execution to use Wave containers. ::: ## Create a personal access token (PAT) If your organization enabled 2FA for your organization or project, you must create a PAT to access your GitLab container registry from Wave. For more information, see [Create a personal access token][gitlab-create]. If your organization created a [project access token][gitlab-project] or a [group access token][gitlab-group], ask your GitLab administrator for access. ## Add credentials to Seqera 1. Add your credentials to your organization or personal workspace: - From an organization workspace: Go to **Credentials > Add Credentials**. - From your personal workspace: From the user menu, go to **Your credentials > Add credentials**. 2. Complete the following fields: - **Name**: Specify a unique name for the credentials using alphanumeric characters, dashes, or underscores. For example, `my-registry-creds`. - **Provider**: Select **Container registry**. - **User name**: Specify your GitLab username. - **Password**: Specify your personal access token (PAT), group access token, or project access token if 2FA is enabled by your GitLab organization. Otherwise specify your GitLab password. - **Registry server**: Specify your GitLab container registry URL. For example, `gitlab.example.com`. 3. After you've completed all the form fields, select **Add**. The new credential is now listed under the **Credentials** tab. [gitlab-cr]: https://docs.gitlab.com/ee/user/packages/container_registry/authenticate_with_container_registry.html [gitlab-pat]: https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html [gitlab-create]: https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html#create-a-personal-access-token [gitlab-project]: https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html [gitlab-group]: https://docs.gitlab.com/ee/user/group/settings/group_access_tokens.html --- ## Google registry credentials From version 22.3, Seqera Platform supports the configuration of credentials for the Nextflow Wave container service to authenticate to private and public container registries. For more information on Wave containers, see the [Nextflow documentation](https://docs.seqera.io/nextflow/wave). :::note Container registry credentials are only used by the Wave container service. Add `wave { enabled=true }` to the **Nextflow config** field on the launch page, or to your `nextflow.config` file, for your pipeline execution to use Wave containers. ::: ## Google Cloud registry access :::note Although Google Cloud Container Registry is still available and supported as a [Google Enterprise API](https://cloud.google.com/blog/topics/inside-google-cloud/new-api-stability-tenets-govern-google-enterprise-apis), new features will only be available in Artifact Registry. Container Registry will only receive critical security fixes. Google recommends using Artifact Registry for all new registries moving forward. ::: Google Cloud Artifact Registry and Container Registry are fully integrated with Google Cloud services and support various authentication methods. Seqera requires programmatic access to your private registry using [long-lived service account keys](https://cloud.google.com/artifact-registry/docs/docker/authentication#json-key) in JSON format. Create dedicated service account keys that are only used to interact with your repositories. Seqera requires the [Artifact Registry Reader](https://cloud.google.com/artifact-registry/docs/access-control#permissions) or [Storage Object Viewer](https://cloud.google.com/container-registry/docs/access-control#permissions) role. ## Create a Google service account with registry access **Google Cloud Artifact Registry** Administrators can create a service account from the Google Cloud console: 1. Go to the [Create service account](https://console.cloud.google.com/projectselector/iam-admin/serviceaccounts/create?walkthrough_id=iam--create-service-account) page. 2. Select a Cloud project. 3. Enter a service account name and (optional) description. 4. Select **Create and continue**. 5. From the **Role** drop-down under step 2, select **Artifact Registry > Artifact Registry Reader**, then select **Continue**. 6. (Optional) Grant other users and admins access to this service account. 7. Select **Done**. 8. From the project service accounts page, select the three dots menu icon under **Actions** for the service account you just created, then select **Manage keys**. 9. On the **Keys** page, select **Add key**. 10. On the **Create private key** popup, select **JSON** and then **Create**. This triggers a download of a JSON file containing the service account private key and service account details. 11. Base-64 encode the contents of the JSON key file: ```bash #Linux base64 KEY-FILE-NAME > NEW-KEY-FILE-NAME #macOS base64 -i KEY-FILE-NAME -o NEW-KEY-FILE-NAME #Windows Base64.exe -e KEY-FILE-NAME > NEW-KEY-FILE-NAME ``` **Google Cloud Container Registry** Administrators can create a service account from the Google Cloud console: 1. Navigate to the [Create service account](https://console.cloud.google.com/projectselector/iam-admin/serviceaccounts/create?walkthrough_id=iam--create-service-account) page. 2. Select a Cloud project. 3. Enter a service account name and an optional description. 4. Select **Create and continue**. 5. From the **Role** drop-down under step 2, search for and select **Storage Object Viewer**, then select **Continue**. 6. (Optional) Grant other users and admins access to this service account under step 3. 7. Select **Done**. 8. From the project service accounts page, select the three dots menu icon under **Actions** for the service account you just created, then select **Manage keys**. 9. On the **Keys** page, select **Add key**. 10. On the **Create private key** popup, select **JSON** and then **Create**. This triggers a download of a JSON file containing the service account private key and service account details. ## Add credentials to Seqera 1. Add your credentials to your organization or personal workspace: - From an organization workspace: Go to **Credentials > Add Credentials**. - From your personal workspace: From the user menu, go to **Your credentials > Add credentials**. 2. Complete the following fields: - **Name**: Specify a unique name for the credentials using alphanumeric characters, dashes, or underscores. For example, `my-registry-creds`. - **Provider**: Select **Container registry**. - **User name**: Specify the service account key type: - Container registry: `_json_key` - Artifact Registry: `_json_key_base64` - **Password**: Specify the JSON key file content. This content is base64-encoded for Artifact Registry. You must remove any line breaks or trailing spaces. For example, `wewogICJ02...9tIgp9Cg==`. - **Registry server**: Specify the container registry hostname, excluding the protocol. For example, `-docker.pkg.dev`. 3. After you've completed all the form fields, select **Add**. The new credential is now listed under the **Credentials** tab. --- ## Managed identities Managed identities offer significant advantages for high performance computing (HPC) environments by enabling granular access control for individual workspace users. Unlike traditional SSH credentials that grant all workspace users access to HPC clusters using the same set of credentials, managed identities ensure each user’s activity is logged under their own credentials. This preservation of user identity is crucial as it naturally inherits the HPC system's fair usage queue policies, mitigates the noisy neighbor problem, and reduces the long wait times associated with First-In-First-Out (FIFO) queues common with shared SSH credentials. Traditional SSH credentials, while simplifying access control to computing resources, result in all user activities on the HPC cluster being logged under the same user credentials. This means all Seqera workspace users have the same access permissions on your HPC cluster, leading to indistinguishable user activities. Managed identities resolve these limitations by allowing administrators to configure a managed identity at the organizational level for access to supported HPC compute environments. This managed identity is selected for authentication similarly to traditional credentials, but contains multiple user credentials each tied to a unique Seqera user. This setup preserves the identity of the user launching workflows on the compute environment and improves traceability and adherence to data access policies. Moreover, with managed identities, users only have the access permissions that their system administrators have granted, minimizing the risk of unauthorized read/write operations in restricted folders. In contrast, shared SSH credentials provide all workspace users with the same access level on the HPC side, which is often more extensive than what an individual user typically needs. By grouping individual user SSH credentials into a single element, managed identities allow administrators to streamline user login and compute environment access while maintaining visibility into data access and compute resource usage for each user. ## Create a managed identity Organization owners can create managed identities at the organization level. A managed identity with user credentials can be used as a credential in HPC clusters for the same provider. 1. From your organization page, select the **Managed identities** tab, then **Add managed identity**. 1. Enter the details of your cluster: - A unique **Cluster name** of your choice using alphanumeric, dash, and underscore characters. - Select a cluster **Provider** from the drop-down. - The fully qualified cluster **Hostname** to be used to connect to the cluster via SSH. This is usually the cluster login node. - The SSH **Port** number for the login connection. The default is port 22. 1. Select **Add cluster**. The new cluster is now listed under your organization's managed identities. Select **Edit** next to a managed identity in the list to edit its details and add user credentials. :::note If the managed identity is already in use on a compute environment, editing its details may lead to errors when using the compute environment. ::: ## Add user credentials Organization owners can grant individual users access to managed identities by adding each user's credentials to the managed identity. You must add user credentials to a managed identity before it can be used in a compute environment. Organization members can add, edit, and delete their own user credentials in a managed identity. :::caution All managed identity users must be a part of the same Linux user group. The group must have access to the HPC compute environment work directory. Set group permissions for the work directory as follows (replace `sharedgroupname` and `` with your group name and work directory): ```bash chgrp -R sharedgroupname chmod -R g+wxs setfacl -Rdm g::rwX ``` These commands change the group ownership of all files and directories in the work directory to `sharedgroupname`, ensure new files inherit the directory's group, and apply default ACL entries to allow the group read, write, and execute permissions for new files and directories. This setup facilitates shared access and consistent permissions management in the directory. ::: 1. From the **Managed identities** tab, select **Edit** next to the cluster in question, then select the **Users** tab. 1. The members of the organization are prepopulated in the **Users** list. Users without credentials are listed with a **Missing** credentials status. Add a user's credentials by selecting **Add credentials** from the user action menu, or the **Add credentials** button. 1. Enter the credential details in the **Add credentials** window: - The member's **Linux username** used to access the cluster. - Paste the contents of the **SSH private key** file for the user's SSH key pair, including the `-----BEGIN OPENSSH PRIVATE KEY-----` and `-----END OPENSSH PRIVATE KEY-----` lines. Ensure no additional lines or spaces are included. - The SSH private key **Passphrase**, if the key has a passphrase. Otherwise, leave this blank. 1. Select **Add credentials**. The Linux username for the user is now populated in the list, and the **Credentials** status is changed to **Added**. Edit existing user credentials by selecting **Edit credentials** from the **Actions** menu next to a user name in the list. --- ## Credentials overview Configure workspace credentials in Seqera Platform to store the access keys and tokens for your [compute environments][compute], [data repositories][data], and [Git hosting services][git]. You can also authenticate AWS and Google Cloud credentials with [workload identity federation][workload]. Platform stores only a role reference, with no long-lived secret. From version 22.3, you can configure container registry credentials that the [Wave container service][wave] uses to authenticate to private and public container registries, such as Docker Hub, Google Artifact Registry, and Quay. For registry-specific instructions, see the Container registry credentials pages. :::note Seqera Platform encrypts all credentials with AES-256 encryption before storing them. No Seqera API exposes credentials in an unencrypted way. ::: ## Credential deletion :::note This section applies only to cloud provider credentials. ::: :::caution Credential deletion is permanent. Past runs on the compute environments that used the credential lose their logs and their work directory view. ::: Deleting a credential has the following effects: - **Platform erases the stored secret.** To restore access, create a new credential and enter the access keys again. - **Platform does not revoke the credential at your cloud provider.** The access key, service account key, or storage key remains valid. To revoke access, also delete or disable the key in your provider console. - **Compute environments that use the credential become invalid.** Platform sets their status message to `Associated credentials have been deleted`. You cannot launch runs on them. - **Platform marks data links that use the credential as `INVALID`.** - **Platform cancels active runs and Studios that use the credential.** To detect them before you delete, call the API with `checked=true`. Platform returns `409` with the list of conflicts instead of deleting the credential: ```bash curl -X DELETE "$TOWER_API_ENDPOINT/credentials/?workspaceId=&checked=true" \ -H "Authorization: Bearer $TOWER_ACCESS_TOKEN" ``` - **Past runs lose their logs and their work directory view.** After deletion, the Nextflow log, the task logs, and the Data Explorer view of the work directory fail for every completed run on that compute environment. Platform does not delete your run data. The data remains in your cloud storage. Only access through Platform is lost. ### Role-based and federated credentials AWS credentials that use an assume-role Amazon Resource Name (ARN), and AWS or Google Cloud credentials that use [workload identity federation][workload], hold no long-lived secret. They store only a role reference: a role ARN for AWS, or a workload identity provider path and service account email for Google Cloud. To revoke access, change the role's trust policy or delete the role. :::note The template uses the `aws` partition. In GovCloud or China, replace `arn:aws:` with `arn:aws-us-gov:` or `arn:aws-cn:` in both the `Federated` principal and the role ARN. The partition must match the one your role is in. ::: ### Restore a compute environment To repair a compute environment in the `Associated credentials have been deleted` state, point it at a replacement credential. You cannot make this change in the Platform UI. Use the API: ```bash curl -X PUT "$TOWER_API_ENDPOINT/compute-envs/?workspaceId=" \ -H "Authorization: Bearer $TOWER_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{"credentialsId":""}' ``` Platform revalidates the compute environment against the replacement credential and returns it to the `AVAILABLE` state. This restores log and work directory access for past runs. Note the following constraints: - Use a credential for the same cloud provider, with access to the same buckets and compute resources. Platform rejects the change if validation fails. - You can only repair a compute environment while it still references the deleted credential. After you attach a working credential, Platform rejects further credential changes. - You cannot update compute environments that authenticate with a [managed identity][managed] this way. {/* links */} [compute]: ../compute-envs/overview [data]: ../data/data-explorer [git]: ../git/overview [wave]: https://docs.seqera.io/wave/provisioning [managed]: ./managed_identities [workload]: ./workload_identity --- ## Quay container registry credentials From version 22.3, Seqera Platform supports the configuration of credentials for the Nextflow Wave container service to authenticate to private and public container registries. For more information on Wave containers, see the [Nextflow documentation](https://docs.seqera.io/nextflow/wave). :::note Container registry credentials are only used by the Wave container service. Add `wave { enabled=true }` to the **Nextflow config** field on the launch page, or to your `nextflow.config` file, for your pipeline execution to use Wave containers. ::: For Quay repositories, we recommend using [robot accounts](https://docs.quay.io/glossary/robot-accounts.html) with **Read** access permissions for authentication. **Create a Quay robot account** 1. Sign in to [quay.io](https://quay.io/). 2. From the user or organization view, select the **Robot Accounts** tab. 3. Select **Create Robot Account**. 4. Enter a robot account name. The username for robot accounts have the format `namespace+accountname`, where `namespace` is the user or organization name and `accountname` is your chosen robot account name. 5. Grant the robot account repository **Read** permissions from **Settings > User and Robot Permissions** in the repository view. 6. Select the robot account in your admin panel to retrieve the token value. ## Add credentials to Seqera 1. Add your credentials to your organization or personal workspace: - From an organization workspace: Go to **Credentials > Add Credentials**. - From your personal workspace: From the user menu, go to **Your credentials > Add credentials**. 2. Complete the following fields: - **Name**: Specify a unique name for the credentials using alphanumeric characters, dashes, or underscores. For example, `my-registry-creds`. - **Provider**: Select **Container registry**. - **User name**: Specify your robot account username. For example, `namespace+accountname`. - **Password**: Specify your robot account access token. For example, `PasswordFromQuayAdminPanel`. - **Registry server**: Specify your container registry hostname. For example, `quay.io`. 3. After you've completed all the form fields, select **Add**. The new credential is now listed under the **Credentials** tab. --- ## SSH credentials SSH public key authentication relies on asymmetric cryptography to generate a public and private key pair. The public key remains on the target (remote) machine, while the private key (and passphrase) is stored in Seqera Platform as a credential. The key pair is used to authenticate a connection with your SSH-enabled environment. To preserve individual user identities by using multiple user SSH credentials to access your HPC compute environments, see [Managed identities](./managed_identities). :::note All credentials are (AES-256) encrypted before secure storage and not exposed in an unencrypted way by any Seqera API. ::: ## Create an SSH key pair To use SSH public key authentication: - The remote system must have a version of SSH installed. This guide assumes the remote system uses OpenSSH. If you're using a different version of SSH, the key generation steps may differ. - The SSH public key must be present on the remote system (usually in `~/.ssh/authorized_keys`). To generate an SSH key pair: 1. From the target machine, open a terminal window and run `ssh-keygen`. 2. Follow the prompts to: - Specify a file path and name (or keep the default). - Specify a passphrase (recommended). 3. Navigate to the target folder (default `/home/user/.ssh/id_rsa`) and open the private key file with a plain text editor. 4. Copy the private key file contents before navigating to Seqera. ## Create an SSH credential in Seqera 1. Add your credentials to your organization or personal workspace: - From an organization workspace: Go to **Credentials > Add Credentials**. - From your personal workspace: From the user menu, go to **Your credentials > Add credentials**. 2. Complete the following fields: - **Name**: A unique name for the credentials using alphanumeric characters, dashes, or underscores. For example, `my-ssh-creds`. - **Provider**: Select **SSH**. - **SSH private key**: Paste the SSH private key file contents. Include the `-----BEGIN OPENSSH PRIVATE KEY-----` and `-----END OPENSSH PRIVATE KEY-----` lines. - **Passphrase**: The SSH private key passphrase (recommended). If your key pair was created without a passphrase, leave this blank. 3. After you've completed all the form fields, select **Add**. The new credential is now listed under the **Credentials** tab. --- ## Workload identity federation With workload identity federation, Seqera Platform connects to your cloud provider account without storing a long-lived workspace credential. Seqera Platform mints a short-lived, signed OpenID Connect (OIDC) token that names your organization, your workspace, and the type of work in progress. Your cloud provider exchanges that token for temporary credentials against a role you control. Workload identity federation is an authentication mode you select when you create an AWS or Google Cloud credential. It is not a separate credential type. Seqera Platform stores only a role reference. For AWS, that is an Identity and Access Management (IAM) role Amazon Resource Name (ARN). For Google Cloud, it is a workload identity provider path and a service account email. You have no access key, service account key, or secret to rotate. :::info Workload identity federation is not available for Azure credentials. See [Limitations](#limitations). ::: ## Enable workload identity federation Workload identity federation requires Seqera Platform Enterprise 26.2 or later, and is enabled in every organization workspace by default. To use it, configure your instance as follows: - Set `TOWER_OIDC_PEM_PATH` to the path of a PEM file that holds an RSA keypair. This variable turns on the Seqera Platform OIDC provider. If it is unset, Seqera Platform serves no JSON Web Key Set (JWKS) endpoint, and no token exchange can complete. See [Cryptographic options][crypto-options]. - Set `TOWER_OIDC_REGISTRATION_INITIAL_ACCESS_TOKEN` to a random value. Setting `TOWER_OIDC_PEM_PATH` also opens the OIDC client registration endpoint. Without this token, anyone who can reach the API can register a client. See [Data features][data-features]. - Serve Seqera Platform over public HTTPS. AWS and Google Cloud fetch `{issuer}/.well-known/openid-configuration` and `{issuer}/.well-known/jwks.json` directly, and fail if they cannot reach the host. Workload identity federation does not work on `localhost`. - Optionally, set `TOWER_IDENTITY_FEDERATION_ALLOWED_WORKSPACES` to a comma-separated list of workspace IDs to restrict workload identity federation to those workspaces. If you omit the variable or leave it empty, every workspace can use it. To generate the keypair: ```bash openssl genrsa -out private.pem 4096 openssl rsa -in private.pem -outform PEM -pubout -out public.pem cat private.pem public.pem > oidc.pem ``` Platform signs every federation token with RS256 using this keypair, and sets the audience your cloud provider expects. `TOWER_AUTH_TOKEN_SIGNING_RS256_ENABLED`, `TOWER_OIDC_ACCESS_TOKEN_AUDIENCE`, and `TOWER_OIDC_AUDIENCE_ENFORCEMENT_ENABLED` apply only to the session and access tokens Platform issues for its own API. Workload identity federation does not need them. :::info Personal workspaces cannot use the workload identity federation credential mode, regardless of configuration. ::: ## Token exchange For each call, Platform mints a short-lived token and exchanges it with your cloud provider for temporary credentials. Your trust policy and permissions decide what those credentials can do. Write your policies against the values in this section. [Configure AWS](#configure-aws) and [Configure Google Cloud](#configure-google-cloud) show one working setup. ### AWS Platform calls `sts:AssumeRoleWithWebIdentity` with the following values: | Field | Value | | --- | --- | | `RoleArn` | The role ARN in the credential | | `RoleSessionName` | `seqera-{workload}`, for example `seqera-data` or `seqera-studio` | | `WebIdentityToken` | A JSON Web Token (JWT) that Platform signs with RS256. It is valid for 300 seconds by default. | The token carries the following claims: | Claim | Value | Present | | --- | --- | --- | | `iss` | Your Platform URL followed by `/api`, for example `https://seqera.example.com/api` | Always | | `sub` | The subject. See [Subjects and attribution](#subjects-and-attribution). | Always | | `aud` | `sts.amazonaws.com` | Always | | `iat`, `exp`, `jti` | Issue time, expiry, and a unique token ID | Always | | `principal_id` | The acting user's ID | When a user is acting | | `principal_email` | The acting user's email address | When a user is acting and has an address | | `https://aws.amazon.com/tags` | The session tags below, all marked transitive | Always | | `https://aws.amazon.com/source_identity` | The acting user's email address, or their ID when the address does not meet AWS's source identity rules | When a user is acting | AWS turns the tags claim into these session tags: | Tag | Value | Present | | --- | --- | --- | | `seqera:org` | Organization ID | Organization workspaces | | `seqera:workspace` | Workspace ID | Organization workspaces | | `seqera:principal-id` | The acting user's ID | When a user is acting | | `seqera:principal-email` | The acting user's email address | When a user is acting and the address fits AWS's tag character set | | `seqera:workload` | `platform`, `data`, `studio`, or `workflow` | Always | Credential validation sends the fixed source identity `seqera-validation` instead of a user. When you save the credential, this checks that the trust policy allows `sts:SetSourceIdentity`. Without the check, a credential could pass validation and then fail on the first call that names a user. ### Google Cloud Platform exchanges the token with the Google Cloud Security Token Service for a federated token. It then impersonates the service account with `generateAccessToken` and requests the `https://www.googleapis.com/auth/cloud-platform` scope. The token carries the same `iss`, `sub`, `iat`, `exp`, and `jti` claims as on AWS, plus `principal_id` and `principal_email` when a user is acting. Its `aud` is the provider resource name, `//iam.googleapis.com/projects/{PROJECT_NUMBER}/locations/global/workloadIdentityPools/{POOL}/providers/{PROVIDER}`, unless you set a custom token audience in the credential. Google Cloud has no session tags or source identity. The acting user reaches Google Cloud Audit Logs only through the `google.subject` mapping. See [Attribute mapping][attribute-mapping]. For a workspace that is not in `TOWER_IDENTITY_FEDERATION_ALLOWED_WORKSPACES`, Platform sends the legacy `workflow` subject on every Google Cloud call. Existing trust configurations keep matching. A change to the allow list takes effect when Platform's cached client for the credential expires. ## Subjects and attribution Every token Seqera Platform generates carries a subject (`sub`) that names the tenant and the kind of work making the request, in the form `org:{orgId}:wsp:{workspaceId}:{workload}`. The trailing segment is the workload type. The following table shows the subject each call presents, and who it is attributed to in `principal_id`, the user session tags, and the source identity. Platform makes every call except a Studio's own mounts and SDK calls, which the Studio container makes through its own token exchange. | Call | Subject | Attributed to | | --- | --- | --- | | Data Explorer bucket list, cached and refreshed in the background | `data` | Unattributed | | Data Explorer browsing, previews, downloads, and uploads | `data` | The browsing user | | Studio mount dialog bucket list | `data` | Unattributed | | Studio mount dialog browsing | `data` | The browsing user | | Studio checkpoints and data-link cache refresh | `data` | Unattributed | | A shared Studio's mounts and SDK calls | `studio` | Unattributed | | A private Studio's mounts and SDK calls | `studio` | The allow-listed user, or the creator if the allow list is empty. Unattributed if more than one user is allowed. | | Credential validation | `platform` | Unattributed. On AWS, the source identity is `seqera-validation`. | | Compute environment describe and provisioning, Forge, job submission, log reads, and Secrets Manager | `platform` | Unattributed | | Pipeline launch bucket probe | `workflow` | The launching user | | The pipeline run itself | Not used | See [Pipeline runs](#pipeline-runs) and [Pipeline runs on Google Cloud](#pipeline-runs-on-google-cloud). | Platform decides a private Studio's attribution each time the Studio starts, because its privacy and allow list can change between sessions. One credential presents all four subjects. Write your trust policy and IAM bindings to admit all of them. A condition that matches only one subject breaks every call that presents another. The acting user is not part of the subject for organization workspaces, because a trust policy is scoped to a workspace. Per-user information travels separately, in session tags and source identity. See [Cloud audit attribution][cloud-audit-attribution]. ## Configure AWS :::info[**Prerequisites**] You need the following: - An AWS account with permission to create IAM identity providers and roles. - A Seqera Platform organization workspace with workload identity federation enabled. ::: Under **Credentials** > **AWS** > **Workload identity**, Seqera Platform shows the values to copy into AWS: the issuer, the four subjects, and the session tag keys. 1. In the AWS console, go to **IAM > Identity providers > Add provider > OpenID Connect**. 1. Set **Provider URL** to `${TOWER_SERVER_URL}/api` and **Audience** to `sts.amazonaws.com`. 1. Select **Create role > Web identity**, then select the provider you created. 1. Replace the trust policy with the template in [Trust policy](#trust-policy). 1. Attach an inline permission policy. See [Permission policies](#permission-policies). 1. In Seqera Platform, create an AWS credential, select **Workload identity**, and enter the role ARN. ### Trust policy Replace the placeholders in this template: - `{{ACCOUNT_ID}}`: Your AWS account ID. - `{{ISSUER_HOST}}`: Your Platform host and its `/api` path, without the scheme. For example, `seqera.example.com/api`. - `{{ORG_ID}}` and `{{WORKSPACE_ID}}`: The values Platform shows in the credential form. The template wildcards the workload segment of the subject, so that one role serves every subject: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::{{ACCOUNT_ID}}:oidc-provider/{{ISSUER_HOST}}" }, "Action": [ "sts:AssumeRoleWithWebIdentity", "sts:TagSession", "sts:SetSourceIdentity" ], "Condition": { "StringEquals": { "{{ISSUER_HOST}}:aud": "sts.amazonaws.com" }, "StringLike": { "{{ISSUER_HOST}}:sub": "org:{{ORG_ID}}:wsp:{{WORKSPACE_ID}}:*" } } } ] } ``` :::note The template uses the `aws` partition. In GovCloud or China, replace `arn:aws:` with `arn:aws-us-gov:` or `arn:aws-cn:` in both the `Federated` principal and the role ARN. The partition must match the one your role is in. ::: Each part of the trust policy is required: - `sts:AssumeRoleWithWebIdentity` performs the exchange. - `sts:TagSession` is required because every token carries session tags. Without it, AWS rejects the entire exchange rather than dropping the tags. - `sts:SetSourceIdentity` is required because tokens for a user's actions carry a source identity, and credential validation tests for it. Without it, AWS rejects the entire exchange. - The `aud` condition accepts only tokens minted for the AWS Security Token Service (STS). - The tenant prefix in the `sub` condition is the security boundary. Without it, any organization or workspace on the same installation could assume the role. The trailing `*` admits every workload type, because one role serves all four. To let one role serve several workspaces, list each workspace's prefix in the condition. ### Permission policies Grant the role the combined permissions of every workload it serves. Because the trust policy's tenant prefix keeps other organizations and workspaces out, the permission policy does not need to repeat it. Credential validation needs nothing beyond STS. A credential can pass validation and still have no data access. | Workload | Needs | | --- | --- | | `platform` | Describe and job submission permissions for your compute environment, `s3:ListBucket` and `s3:GetObject` on the work directory, and the [Forge permissions](#batch-forge-and-cloud-forge) if Forge creates the environment | | `data` | `s3:ListAllMyBuckets`, plus object access on the buckets Data Explorer shows and on the work directory, where Studio checkpoints are stored | | `studio` | Object access on the buckets your Studios mount and on the work directory | | `workflow` | `s3:ListBucket` on the work directory and the compute environment's allowed buckets, for the [launch probe](#pipeline-launch-bucket-probe) | Grant bucket discovery in its own statement. Because `s3:ListAllMyBuckets` has no resource dimension, a denial fails the whole listing instead of returning fewer buckets: ```json { "Sid": "ListBuckets", "Effect": "Allow", "Action": "s3:ListAllMyBuckets", "Resource": "*" } ``` Grant object access on each bucket Data Explorer shows, each bucket your Studios mount, and the work directory: ```json { "Sid": "Buckets", "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketAcl", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload" ], "Resource": [ "arn:aws:s3:::BUCKET", "arn:aws:s3:::BUCKET/*" ] } ``` Data Explorer uses `s3:GetBucketAcl` only to mark public buckets. Without it, every bucket shows as private. :::caution When the trust policy is correct but the permission policy has no matching statement, the exchange succeeds and AWS denies every request afterwards. See [A Studio starts but its data does not mount][wif-studio-mount]. ::: For compute environments, grant the describe permissions: ```json { "Sid": "PlatformDescribe", "Effect": "Allow", "Action": [ "ec2:Describe*", "batch:Describe*", "batch:List*", "ecs:Describe*", "ecs:List*", "eks:DescribeCluster", "eks:ListClusters", "iam:ListRoles", "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:GetLogEvents" ], "Resource": "*" } ``` When you create an AWS Batch compute environment, Platform checks that the work directory is in the environment's region. This check needs `s3:ListBucket`. Platform also reads task logs and files from the work directory, which needs `s3:GetObject`. The `Buckets` statement covers both if it includes the work directory. To submit runs, add `batch:RegisterJobDefinition`, `batch:SubmitJob`, `batch:TerminateJob`, `batch:TagResource`, and `iam:PassRole`. Platform registers a job definition on the first launch that has no matching one. A role without `batch:RegisterJobDefinition` fails on its first run. AWS Batch compute environments need no EC2 launch permissions, because Batch scales the environment with its own service role. AWS Cloud compute environments launch and terminate EC2 instances directly, under the `platform` subject. For AWS Cloud, also grant the role the EC2 permissions listed in [AWS Cloud][aws-cloud-permissions]. #### Narrow access by workload or user The permissions above apply to every workload the role serves. To restrict a statement to one workload type, add a condition on the `seqera:workload` session tag, for example `"StringEquals": {"aws:PrincipalTag/seqera:workload": "data"}`. The `{{ISSUER_HOST}}:sub` condition key works too, because AWS keeps the `sub` claim available for the whole session. To restrict a statement by user, condition it on `seqera:principal-id`. See [Per-user access control](#per-user-access-control). ### Batch Forge and Cloud Forge Batch Forge and Cloud Forge run under workload identity and present the `platform` subject. Grant IAM write permissions scoped to the `TowerForge-*` name prefix, and the compute permissions Forge uses to build the environment: ```json { "Sid": "Forge", "Effect": "Allow", "Action": [ "iam:CreateRole", "iam:DeleteRole", "iam:GetRole", "iam:TagRole", "iam:PassRole", "iam:AttachRolePolicy", "iam:DetachRolePolicy", "iam:ListAttachedRolePolicies", "iam:PutRolePolicy", "iam:DeleteRolePolicy", "iam:ListRolePolicies", "iam:CreateInstanceProfile", "iam:DeleteInstanceProfile", "iam:GetInstanceProfile", "iam:TagInstanceProfile", "iam:AddRoleToInstanceProfile", "iam:RemoveRoleFromInstanceProfile" ], "Resource": [ "arn:aws:iam::{{ACCOUNT_ID}}:role/TowerForge-*", "arn:aws:iam::{{ACCOUNT_ID}}:instance-profile/TowerForge-*" ] }, { "Sid": "ForgeCompute", "Effect": "Allow", "Action": [ "batch:*ComputeEnvironment", "batch:*JobQueue", "batch:Describe*", "batch:TagResource", "ec2:CreateLaunchTemplate", "ec2:DeleteLaunchTemplate", "ec2:Describe*", "ssm:GetParameters", "elasticfilesystem:*", "fsx:*" ], "Resource": "*" } ``` :::caution Keep the resource scope. A principal that can call both `iam:CreateRole` and `iam:PassRole` without a resource scope can create a role more privileged than itself and then pass it. Forge names what it creates `{prefix}-{id}-{RoleKind}`, where the prefix defaults to `TowerForge` and is set by `TOWER_FORGE_PREFIX`. If your deployment overrides that variable, scope the policy to your own prefix instead. A policy scoped to `TowerForge-*` denies every Forge role creation on an installation that renamed it. ::: Remove `elasticfilesystem:*` and `fsx:*` if the environment mounts neither. Cloud Forge needs the `Forge` statement and only the `ec2` actions from `ForgeCompute`. ### Pipeline runs Workload identity federation authenticates the calls Platform makes itself, for compute environment setup, job submission, Data Explorer, and Studios. A pipeline run does not use it. The Nextflow head job and every task it launches read the EC2 instance role from instance metadata. The workload identity role plays no part once the job starts. Before the launch, Platform checks that the run's buckets are reachable, under the `workflow` subject and attributed to the launching user. See [Pipeline launch bucket probe](#pipeline-launch-bucket-probe). The instance role needs the permissions Nextflow uses. On AWS Batch, these are S3 access on the work-directory bucket, plus Batch, ECS, EC2, and CloudWatch Logs permissions. Do not condition these grants on `:sub`, because an instance-profile session presents no OIDC subject. On a compute environment that Forge creates, Forge writes these grants. On one you create manually, add them to the instance role yourself. :::caution Leave `TOWER_WIF_FORGE_LEGACY_MODE_ENABLED` at its default, `true`. With `false`, Forge grants the instance role no data access. A compute environment that Forge creates for a workload identity credential then cannot run a pipeline, because the head job starts with no credentials. Studios and compute environment provisioning are unaffected. The variable applies to the whole installation. Forge reads it when it creates a compute environment. Changing it does not affect existing ones. ::: ## Configure Google Cloud :::info[**Prerequisites**] You need the following: - A Google Cloud project with permission to create workload identity pools and service accounts. - A Seqera Platform organization workspace with workload identity federation enabled. ::: Under **Credentials** > **Google** > **Workload Identity**, Seqera Platform shows the values to copy into Google Cloud: the OIDC issuer URL, the `google.subject` mapping, and the recommended attribute condition. :::note Platform uses the project in the **Workload identity provider** path, which is the project that hosts the pool, as the credential's project. Credential validation and Data Explorer list buckets in that project. Compute environments run their jobs and VMs there, and Platform creates pipeline secrets and reads Cloud Logging there. The service account can live in another project, but it needs its roles on the pool's project. Google recommends keeping pools in a [dedicated project][gcp-wif-dedicated-project]. If you follow that recommendation, these calls also go to the dedicated project. ::: 1. In the project that hosts the pool, enable the IAM, Resource Manager, Service Account Credentials, and Security Token Service APIs. See [Configure Workload Identity Federation][gcp-wif-configure]. 1. In the Google Cloud console, go to the **New workload provider and pool** page. Under **Create an identity pool**, enter a **Name** and **Description**, then select **Continue**. The name is also the pool ID, and you can't change it later. 1. Under **Configure provider settings**, in **Select a provider**, select **OpenID Connect (OIDC)**. Enter a **Provider name**, which is also the provider ID, and set **Issuer URL** to `${TOWER_SERVER_URL}/api`. See [Create a workload identity pool and provider][gcp-wif-pool]. 1. Under **Audiences**, keep **Default audience**. The console shows it as `https://iam.googleapis.com/projects/{PROJECT_NUMBER}/locations/global/workloadIdentityPools/{POOL}/providers/{PROVIDER}`. Platform sends the same path without the scheme, `//iam.googleapis.com/projects/{PROJECT_NUMBER}/locations/global/workloadIdentityPools/{POOL}/providers/{PROVIDER}`, and the default audience accepts both forms. If you select **Allowed audiences** instead, add the `//iam.googleapis.com/...` form to the list. Select **Continue**. 1. Under **Configure provider attributes**, set the `google.subject` mapping. Under **Attribute conditions**, enter the recommended condition. Select **Save**. See [Attribute mapping][attribute-mapping]. 1. Create or select a service account, and set up the two grants that [credential validation](#credential-validation) needs before you continue. They are on different tabs of the service account's page: - **Permissions** tab, for what the service account can access: select **Manage access** and add Storage Bucket Viewer (`roles/storage.bucketViewer`). This grants the role on the service account's own project. It works only when that project is also the pool's project. Otherwise, grant it on the pool project's **IAM** page, with the service account as the principal. - **Principals with access** tab, for who can act as the service account: select **Grant access**, enter the pool's principal, and add Workload Identity User (`roles/iam.workloadIdentityUser`). Without it, Platform cannot use the service account at all. Don't add this role on the **Permissions** tab or in the create flow. Those grant roles to the service account itself, which does not let the pool act as it. Without both, Platform saves the credential but marks it `INVALID`. IAM changes can take a few minutes to apply. For the principal to enter and the permissions each workload needs, see [Impersonation and permissions][impersonation-and-permissions]. 1. In Seqera Platform, create a Google credential, select **Workload Identity**, and enter: - **Workload identity provider**: The provider resource name, in the form `projects/{PROJECT_NUMBER}/locations/global/workloadIdentityPools/{POOL}/providers/{PROVIDER}`. Use the project number, not the project ID. See [Identifying projects][gcp-project-number]. - **Service account email**: The service account to impersonate, in the form `NAME@PROJECT_ID.iam.gserviceaccount.com`. - **Token audience** (optional): Leave empty. Platform then uses the provider path as the audience. The credential uses workload identity federation only when both **Service account email** and **Workload identity provider** are set. Saving the credential runs [credential validation](#credential-validation). :::caution A pool or provider resource ID is immutable. To change one, create a replacement rather than renaming it. ::: ### Attribute mapping Set the `google.subject` mapping on the OIDC provider: ``` assertion.sub + (has(assertion.principal_id) ? ':usr:' + assertion.principal_id : '') ``` This mapping appends the acting user to the tenant subject. The subject Google records in its audit logs then identifies a specific user, for example `org:{{ORG_ID}}:wsp:{{WORKSPACE_ID}}:data:usr:{{USER_ID}}`. The `has()` guard keeps background tokens valid. Background tokens carry no `principal_id` and fall back to the tenant-only subject. Without the guard, their exchange fails with `Could not obtain a value for google.subject`. See [Mappings and conditions][gcp-wif-mappings]. :::caution Map the user through `google.subject`, not a custom `attribute.user`. Custom attributes work in IAM conditions and `principalSet` bindings, but Google never writes them to audit logs. Only `google.subject` reaches the log. ::: Set the attribute condition Platform recommends in the credential form: ``` assertion.sub.startsWith('org:{{ORG_ID}}:wsp:{{WORKSPACE_ID}}:') ``` Without this condition, a whole-pool impersonation binding accepts subjects minted for any other tenant on the same Platform installation. Use a prefix rather than an exact match, because every workload type must pass, including the `workflow` subject Platform sends for workspaces where federation is not enabled. ### Impersonation and permissions Allow the pool's identities to impersonate the service account. Grant the pool's principal the Workload Identity User role (`roles/iam.workloadIdentityUser`) on the service account itself, not on the project. A project-level grant [applies to every service account in the project][gcp-sa-project-grants]. See [Service account impersonation][gcp-wif-impersonation]. Build the principal from the credential's **Workload identity provider** value. Drop the `/providers/PROVIDER` segment, prefix `principalSet://iam.googleapis.com/`, and end with `/*` for every identity in the pool, or with `/attribute.workspace/WORKSPACE_ID` for one workspace: | | Value | | --- | --- | | **Workload identity provider** | `projects/123456789012/locations/global/workloadIdentityPools/seqera-pool/providers/seqera-oidc` | | Principal for every identity in the pool | `principalSet://iam.googleapis.com/projects/123456789012/locations/global/workloadIdentityPools/seqera-pool/*` | | Principal for one workspace | `principalSet://iam.googleapis.com/projects/123456789012/locations/global/workloadIdentityPools/seqera-pool/attribute.workspace/67890` | The principal uses the pool's project number and pool ID, not the project ID or the pool's display name. See [Principal identifiers][gcp-principal-identifiers]. To grant the role in the Google Cloud console: 1. Go to the **Service Accounts** page, select the service account's project, and select the service account's email address. 1. Open the **Principals with access** tab and select **Grant access**. 1. Enter the principal. 1. Assign the **Workload Identity User** role, then select **Save**. See [Grant a single role][gcp-sa-grant-role]. Service Account User (`roles/iam.serviceAccountUser`) does not work here. It lacks `iam.serviceAccounts.getAccessToken`, and Google denies impersonation without it. To grant the role with the gcloud CLI: ```bash gcloud iam service-accounts add-iam-policy-binding SA_EMAIL \ --role=roles/iam.workloadIdentityUser \ --member="principalSet://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL/*" ``` The whole-pool principal admits every identity that passes the provider's attribute condition. With the recommended condition, that is one workspace. If one pool serves several workspaces, Google recommends [against granting access to all members of a pool][gcp-wif-avoid-all-members]. Add the attribute mapping `attribute.workspace = assertion.sub.extract("wsp:{workspace}:")`, and bind each workspace's service account to its one-workspace principal. One such binding covers all of that workspace's workloads. :::caution Do not bind an exact subject (`principal://.../POOL/subject/SUBJECT`) or an `attribute.workload` value. Both break when a subject changes shape. Use a pool-wide or `attribute.workspace` binding. ::: Platform signs download URLs through the IAM `signBlob` API. This needs `roles/iam.serviceAccountTokenCreator` on the service account, bound to the service account itself: ```bash gcloud iam service-accounts add-iam-policy-binding SA_EMAIL \ --role=roles/iam.serviceAccountTokenCreator \ --member="serviceAccount:SA_EMAIL" ``` Without this binding, viewing or downloading file contents in Data Explorer fails with a signing error. Pipeline runs are unaffected. Grant the service account the combined permissions of every workload it serves. Every workload impersonates the same service account, and Google Cloud evaluates that service account as the principal. Each grant therefore applies to every workload type. | Workload | Needs | | --- | --- | | `platform` | `storage.buckets.list` on the pool's project, for [credential validation](#credential-validation) and the compute environment form. Object read on the work directory, for run logs and reports. The describe, job submission, and log permissions for your compute environment. See [Pipeline runs on Google Cloud](#pipeline-runs-on-google-cloud). | | `data` | `storage.buckets.list` on the pool's project, plus the Data Explorer permissions below on the buckets Data Explorer shows and on the work directory, where Studio checkpoints are stored | | `studio` | Object access on the buckets your Studios mount and on the work directory | | `workflow` | `storage.objects.list` on the work-directory bucket, for the [launch probe](#pipeline-launch-bucket-probe) | For Data Explorer, the service account needs both bucket-level and object-level permissions: | Permission | Needed for | | --- | --- | | `storage.buckets.list` | Listing the buckets in the pool's project that Data Explorer shows | | `storage.buckets.get` | Opening a bucket or adding one to Data Explorer. Platform reads the bucket's metadata before each listing. | | `storage.objects.list`, `storage.objects.get` | Browsing and downloading | | `storage.objects.create` | Uploading | | `storage.objects.delete` | Deleting | Object roles such as `roles/storage.objectViewer` include no bucket permissions. With only object roles, the service account can't list or open buckets in Data Explorer. Grant these two roles: - `roles/storage.bucketViewer` on the pool's project, to list and open buckets. Listing buckets is a project-level action. A grant on a single bucket isn't enough. - `roles/storage.objectAdmin` on each bucket, to browse, download, upload, and delete. Data Explorer discovers buckets in the pool's project only. To browse a bucket in another project, add it with **Add data repository**, and grant the service account `storage.buckets.get` and object access on it. See [Add data repository links][data-explorer-add]. ### Credential validation Platform validates the credential when you save it, and re-checks a valid credential about every 12 hours. The check presents the `platform` subject, or `workflow` in a workspace that `TOWER_IDENTITY_FEDERATION_ALLOWED_WORKSPACES` leaves out. The check runs the whole chain. It exchanges a token with the Security Token Service, impersonates the service account, and lists buckets in the pool's project. It passes only when all of the following are true: - The provider accepts the token. The issuer URL, audience, and attribute condition all match. - The pool's principal holds the Workload Identity User role on the service account. - The service account holds the `storage.buckets.list` permission on the pool's project, for example through `roles/storage.bucketViewer`. Unlike on AWS, where validation needs nothing beyond the token exchange, a Google credential with a correct trust setup still fails validation without `storage.buckets.list` on the pool's project. A failed check marks the credential `INVALID`, with a reason that starts with `Cannot validate Google WIF Security Keys, reason:`. Throttling and Google server errors leave the status unchanged. Platform does not re-check an `INVALID` credential on its own. After you fix the cause, select **Validate** on the credential. See [Validate a credential manually][preflight-validate]. IAM changes typically take 2 minutes to apply, and sometimes 7 minutes or longer. See [Access change propagation][gcp-iam-propagation]. ### Pipeline runs on Google Cloud As on AWS, a pipeline run does not use workload identity federation. The Nextflow head job and its tasks authenticate as the service account attached to their VMs. On Google Cloud Batch, that is the compute environment's **Service account email**. If you leave it empty, Platform sets it to the credential's own service account. The head job and every task then run as that service account. It needs the [Google Cloud Batch service account permissions][gcp-batch-sa-permissions]. Its Service Account User role (`roles/iam.serviceAccountUser`) must also cover itself, because Platform submits the jobs as the same account. Google requires Service Account User on a job's service account to create the job. See [Control access for a job using a custom service account][gcp-batch-custom-sa]. For Platform's own calls under the `platform` subject, the service account also needs the following on the pool's project: - `roles/compute.viewer`, for the zones, machine types, images, and networks in the compute environment form. - `roles/logging.viewer`, to read Google Cloud Batch job logs from Cloud Logging. It includes `resourcemanager.projects.get`, which Platform needs to resolve the project name from its number. - `roles/secretmanager.admin`, if your pipelines use secrets. Platform creates and deletes pipeline secrets in the pool's project. On Google Cloud compute environments, Cloud Forge creates a service account for the VM, under the `platform` subject. With `TOWER_WIF_FORGE_LEGACY_MODE_ENABLED` at its default, `true`, Forge grants it `roles/storage.objectAdmin` on the work-directory bucket, and `roles/logging.logWriter`, `roles/monitoring.metricWriter`, `roles/storage.bucketViewer`, and `roles/storage.objectViewer` on the pool's project. With `false`, Forge grants only `roles/logging.logWriter` and `roles/monitoring.metricWriter`, and the VM has no storage access of its own. The caution in [Pipeline runs](#pipeline-runs) applies. The credential's service account needs the [Google Cloud permissions][gcp-cloud-permissions] that Forge uses to create these resources. ### Existing Google Cloud credentials Before Seqera Platform Enterprise 26.2, a Google workload identity credential always presented the `workflow` subject. From 26.2, the credential presents `platform`, `data`, `studio`, or `workflow`, depending on the request. :::caution If you already use a Google workload identity credential, check your impersonation bindings before you upgrade. Because workload identity federation is on in every organization workspace by default, the credential starts presenting all four subjects when you upgrade. A binding against the exact subject (`principal://.../POOL/subject/org:{{ORG_ID}}:wsp:{{WORKSPACE_ID}}:workflow`) or against an `attribute.workload` value then stops matching, and the workspace loses access. Move those bindings to a pool-wide (`POOL/*`) or `attribute.workspace` form first. Both read the tenant part of the subject, which does not change. To keep a workspace on the `workflow` subject while you move its bindings, set `TOWER_IDENTITY_FEDERATION_ALLOWED_WORKSPACES` to a list that leaves it out. See [Google Cloud](#google-cloud). ::: ## Cloud audit attribution Workload identity sessions carry the acting user's identity into your cloud provider's audit log, where you can attribute a request to a specific Seqera user. The two providers record the identity differently: | | What the audit log records | What it takes | | --- | --- | --- | | **AWS** | The assumed IAM role on every entry, plus the acting user as session tags on the exchange event and as a source identity on every request in the session | Only the trust policy. Platform always attaches the tags, and a source identity whenever a user is acting. The trust policy must grant `sts:TagSession` and `sts:SetSourceIdentity` or the exchange fails outright | | **Google Cloud** | The mapped subject, which carries the acting user only if the `google.subject` mapping appends `principal_id` | The guarded mapping in [Attribute mapping][attribute-mapping], and Data Access audit logs, which are off by default and billed separately | The identifier differs by provider. On AWS, the source identity is the user's email where one is known, and their ID otherwise. On Google Cloud, `principal_id` is always an internal numeric ID. ### AWS Every session carries the session tags listed in [Token exchange](#aws), and a source identity when a user is behind the request. Use tags to authorize a request and source identity to trace it. `aws:PrincipalTag/*` condition keys match tags. CloudTrail records them as `principalTags` on the `AssumeRoleWithWebIdentity` event only, never on the requests made afterwards. CloudTrail records source identity on every request in the session, in `userIdentity.sessionContext.sourceIdentity`. Source identity is the only way to determine who read a given object. Because the role session name is `seqera-{workload}`, CloudTrail shows the workload type in every assumed-role ARN. To scope one role across many workspaces, use the workspace tag as a policy variable: ```json { "Sid": "PerWorkspaceBucket", "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetObject" ], "Resource": [ "arn:aws:s3:::wsp-${aws:PrincipalTag/seqera:workspace}", "arn:aws:s3:::wsp-${aws:PrincipalTag/seqera:workspace}/*" ] } ``` #### Per-user access control Workload identity federation resolves to one role per workspace, not one role per user. Every user in the workspace assumes the same role and, by default, has the same cloud permissions. Per-user access control comes from your IAM policy, not from Platform. Condition a statement on the `seqera:principal-id` tag, and your cloud provider decides what that user can reach. An explicit `Deny` overrides every `Allow`. To deny one person access to a bucket: ```json { "Sid": "BlockUser", "Effect": "Deny", "Action": "s3:*", "Resource": [ "arn:aws:s3:::BUCKET", "arn:aws:s3:::BUCKET/*" ], "Condition": { "StringEquals": { "aws:PrincipalTag/seqera:principal-id": "{{USER_ID}}" } } } ``` :::caution A deny list allows every new user by default. Allow-listing by team is not possible because Platform does not emit team membership as a session tag. Plan your policies around denying named users rather than admitting named teams. A `seqera:principal-id` condition also has no effect inside a Studio shared with the workspace. A shared session carries no acting user. It presents itself as the Studio, with the subject `org:{orgId}:wsp:{workspaceId}:studio`, not as a user. Your policy can match the Studio but not an individual user. To limit what a shared Studio can reach, condition a statement on the `studio` workload and scope its `Resource` to those buckets. See [Narrow access by workload or user](#narrow-access-by-workload-or-user). ::: Note the following when you write policies against these values: - Platform asserts the tag values. AWS trusts the provider and does not verify them. - Condition on `seqera:principal-id`, which is stable. Use `seqera:principal-email` for reading only. Addresses are mutable and reassignable, and absent for service accounts. - A condition on an absent tag does not match. A statement requiring `seqera:principal-id` denies every `platform` session and all background work, such as cache refresh and job polling. - Renaming these keys is a breaking change, because they appear in your IAM policies. ### Google Cloud Google Cloud Audit Logs record only the mapped `google.subject`. Custom claims and attributes never reach the log. Organization, workspace, and workload type are traceable because they are part of the base subject. The acting user is traceable only if the `google.subject` mapping appends `principal_id`. See [Attribute mapping][attribute-mapping]. Cloud Storage reads and writes are Data Access audit logs, which are off by default. Enable them on the project or service account to see Data Explorer activity. Because previews and downloads use a URL signed by the service account, Cloud Storage logs them as the service account. The token exchange and impersonation entries need Data Access **Admin Read** for the Security Token Service and IAM Service Account Credentials APIs. ### Requests with no acting user Background work (data link cache refresh, Studio checkpoints, and job polling) carries no acting user. Neither does a Studio shared with the workspace, because no single person operates a shared session for its whole life. Those entries show the tenant subject with no per-user attribution. ## Pipeline launch bucket probe When the preflight check is enabled, compute environments that use workload identity probe the work directory before a launch. A run that cannot reach its work directory fails immediately instead of part-way through. On AWS, the probe performs a fresh, uncached token exchange with the `workflow` subject, then calls `ListObjectsV2` with `maxKeys=1` against the work directory and each of the compute environment's allowed buckets. Grant `s3:ListBucket` on each of those buckets under the `workflow` subject. On Google Cloud, the probe lists the root of the work-directory bucket rather than the work-directory prefix, and it does not check allowed buckets. Grant `storage.objects.list` on the bucket itself. A grant conditioned on the work-directory prefix is denied. The probe response decides the launch: - An explicit `AccessDenied` refuses the launch with `WORK_DIR_INVALID`. The message names the subject and the bucket. - When AWS refuses the token exchange itself, the launch is also refused. The message names the subject only, because no bucket was reached. A trust policy that does not admit the `workflow` subject fails this way, with `AccessDenied` from the token exchange. - Inconclusive responses (throttling, quota, billing, and timeouts) log a warning and allow the launch. Platform does not probe credentials that use access keys or an assumed role. :::note The Google Cloud probe runs from your Platform instance's network location. VPC Service Controls or organization policies can deny that request even when a Batch job inside the permitted perimeter could reach the bucket. ::: ## Credential revocation Platform briefly caches the temporary cloud credentials it derives from a workload identity credential. Editing or deleting the credential, or removing a user from the workspace, drops the affected cache entries immediately across all nodes. Platform cannot recall access it has already handed out: - On AWS, a presigned URL embeds the temporary credential and stays valid until that credential expires, up to approximately one hour. - On Google Cloud, a presigned URL is signed by the service account and stays valid for the Data Explorer URL duration, one day by default, regardless of the credential's lifetime. - A running Studio keeps renewing its workload identity until it stops, for up to three days by default. Deleting the credential or removing the user does not end a running session's access. To cut it off, stop the Studio, or change the role's trust policy or the service account's impersonation binding. ## Limitations - Azure is not supported. Azure federated identity credentials match the subject claim by exact string. That would require one federated credential per workspace per workload type, against a cap of 20 per identity. Flexible federated identity credentials solve this with wildcard matching, but only for a fixed list of Microsoft-supported issuers that Platform cannot join. - You cannot change an AWS credential's mode after creation. To move an existing AWS credential to workload identity, create a new credential. Google credentials have no mode field, and you can update them in place. - On AWS, Data Explorer omits buckets the credential cannot reach rather than showing them as inaccessible. You cannot distinguish an omitted bucket from one that does not exist. This applies to every AWS credential type, not only workload identity federation. - The credential form accepts only `arn:aws:` role ARNs. To use a role in the `aws-us-gov` or `aws-cn` partition, create the credential through the API. - You can create AWS workload identity credentials through version 1 of the API only. Google workload identity credentials are available in both versions. - [Data lineage][data-lineage] does not support workload identity credentials. Platform builds its lineage bucket and SNS topic clients without a workload identity. Lineage works only with key-based or role-based AWS credentials. - A trust policy error does not prevent credential creation. Saving runs the token exchange, but a failed exchange does not roll back the new credential. Check the credential's status to confirm the exchange succeeded. For token exchange, permission, and audit attribution failures, see [Workload identity troubleshooting][wif-troubleshooting]. [crypto-options]: ../enterprise/configuration/overview#cryptographic-options [data-features]: ../enterprise/configuration/overview#data-features [data-lineage]: ../data/data-lineage [gcp-wif-pool]: https://cloud.google.com/iam/docs/workload-identity-federation-with-other-providers#create-pool-provider [gcp-wif-mappings]: https://cloud.google.com/iam/docs/workload-identity-federation-with-other-providers#mappings-and-conditions [gcp-wif-configure]: https://cloud.google.com/iam/docs/workload-identity-federation-with-other-providers#configure [gcp-wif-impersonation]: https://cloud.google.com/iam/docs/workload-identity-federation-with-other-providers#impersonation [gcp-wif-dedicated-project]: https://cloud.google.com/iam/docs/best-practices-for-using-workload-identity-federation#dedicated-project [gcp-wif-avoid-all-members]: https://cloud.google.com/iam/docs/best-practices-for-using-workload-identity-federation#avoid-all-members [gcp-principal-identifiers]: https://cloud.google.com/iam/docs/principal-identifiers#allow [gcp-project-number]: https://cloud.google.com/resource-manager/docs/view-update-projects#identifying_projects [gcp-sa-project-grants]: https://cloud.google.com/iam/docs/best-practices-service-accounts#project-folder-grants [gcp-sa-grant-role]: https://cloud.google.com/iam/docs/manage-access-service-accounts#grant-single-role [gcp-iam-propagation]: https://cloud.google.com/iam/docs/access-change-propagation [gcp-batch-custom-sa]: https://cloud.google.com/batch/docs/create-run-job-custom-service-account#before-you-begin [gcp-batch-sa-permissions]: ../compute-envs/google-cloud-batch#service-account-permissions [gcp-cloud-permissions]: ../compute-envs/google-cloud#required-permissions [preflight-validate]: ../compute-envs/preflight-checks#validate-a-credential-manually [data-explorer-add]: ../data/data-explorer#add-data-repository-links [aws-cloud-permissions]: ../compute-envs/aws-cloud#required-permissions [wif-troubleshooting]: ../troubleshooting_and_faqs/workload_identity_troubleshooting [wif-studio-mount]: ../troubleshooting_and_faqs/workload_identity_troubleshooting#a-studio-starts-but-its-data-does-not-mount [cloud-audit-attribution]: #cloud-audit-attribution [trust-policy]: #trust-policy [permission-policies]: #permission-policies [attribute-mapping]: #attribute-mapping [impersonation-and-permissions]: #impersonation-and-permissions --- ## Data Explorer With Data Explorer, you can browse and interact with remote data repositories from organization workspaces in Seqera Platform. It supports AWS S3, Azure Blob Storage, Google Cloud Storage, and Amazon S3-compatible API storage (for example, Cloudflare R2, MinIO, and Oracle Cloud). Access the **Data Explorer** tab from any workspace to view and manage all available data repositories. Data Explorer is also integrated with the pipeline launch form, run detail pages, and Studios. Use these integrations to select input data files and output directories, view the output files of a run, or use files in object storage directly for interactive analysis. ## Participant roles The role assigned to a workspace user affects what functionality is available in Data Explorer. These permissions are listed in the [Participant roles][roles]. - **View**: Can view, download, and preview contents of cloud storage buckets. Cannot upload. Cannot hide or add buckets. - **Launch**: Can view, download, and preview contents of cloud storage buckets. Cannot upload. Cannot hide or add buckets. - **Connect**: Can view, download, and preview contents of cloud storage buckets. Cannot upload. Cannot hide or add buckets. - **Project**: Can view, download, and preview contents of cloud storage buckets. Cannot upload. Cannot hide or add buckets. - **Maintain**: Can view, download, upload, and preview contents of cloud storage buckets. Can hide and add buckets. - **Admin**: Can view, download, upload, and preview contents of cloud storage buckets. Can hide and add buckets. - **Owner**: Can view, download, upload, and preview contents of cloud storage buckets. Can hide and add buckets. ## Access control Two mechanisms control Data Explorer access: - **Participant roles** determine which Data Explorer actions a workspace user can perform, such as browsing, previewing, downloading, and uploading. See [Participant roles][roles]. - **Credentials** determine which objects those actions can reach. A manually added data-link uses the credentials you select when you add the data repository to the workspace. A data-link that Data Explorer retrieves automatically uses one of the workspace credentials that can access the repository, without a deterministic selection order. See [Add data repository links](#add-data-repository-links). The cloud provider permissions attached to those credentials define the scope of Data Explorer access to that repository. To narrow what Data Explorer can do in a bucket, assign that data-link a dedicated credential with a more restrictive cloud provider policy. Sharing one broad credential across compute environments and data repositories gives Data Explorer the full scope of that credential. When a data-link uses a [workload identity federation][wif] credential, Data Explorer reaches storage by exchanging a short-lived token for temporary cloud credentials at request time, instead of using a stored key. The role the credential names determines access. No stored key exists whose scope you need to check. If the token exchange fails, the request fails. Data Explorer does not fall back to a shared or stored credential. :::caution Do not condition an IAM policy on the `seqera:principal-id` session tag when that policy governs bucket discovery. Because Data Explorer discovers buckets in a background refresh that carries no acting user, the tag is absent and the condition never matches. The bucket disappears for every workspace member. See [Cloud audit attribution][wif-audit]. ::: Data Explorer has no per-bucket or per-workspace setting that disables downloads or uploads while leaving browsing available. Two instance-level [environment variables](../enterprise/configuration/overview#data-features) control Data Explorer availability: - `TOWER_DATA_EXPLORER_ENABLED` enables or disables Data Explorer for every workspace in your Enterprise instance. This is the only way to remove download and upload access completely. - `TOWER_DATA_EXPLORER_CLOUD_DISABLED_WORKSPACES` disables automatic cloud bucket retrieval in the listed workspaces. This is not a download or upload control. Manually added data-links remain usable in those workspaces. :::warning Cross-origin resource sharing (CORS) is not an access-control mechanism. Browsers enforce CORS, and it covers only the upload and multi-file download paths described in [CORS configurations for cloud providers](#cors-configurations-for-cloud-providers). Leaving a bucket's CORS configuration unset does not prevent Data Explorer users from reaching the objects in that bucket. CORS has no effect on access through the Seqera Platform API, the Seqera Platform CLI (`tw`), or your cloud provider's tools. Use credentials and cloud provider access policies to control access to your data. ::: ## Add data repository links Data Explorer lists public and private data repositories. Repositories accessible to your workspace credentials are retrieved automatically; workspace maintainers can also configure repositories manually. - **Retrieve data repositories with workspace credentials** Data Explorer automatically lists private data repositories that the credentials defined in your workspace can access. The permissions required for your [AWS](../compute-envs/aws-batch#iam-user-creation), [Google Cloud](../compute-envs/google-cloud-batch#iam), [Azure Batch](../compute-envs/azure-batch#storage-account), or high-performance computing (HPC) compute environment credentials allow full Data Explorer functionality. For AWS S3, Data Explorer requires the following minimum IAM permissions: - `s3:ListAllMyBuckets` (on `*`) to auto-discover the buckets accessible to your workspace credentials. - `s3:ListBucket` and `s3:GetBucketAcl` on each bucket you want to browse, to resolve its region and access configuration. - `s3:GetObject`, `s3:PutObject`, `s3:DeleteObject`, and `s3:AbortMultipartUpload` on the objects in each bucket, to download, upload, and delete files. These are a subset of the S3 permissions documented for the [AWS Batch](../compute-envs/aws-batch#required-platform-iam-permissions), [AWS Cloud](../compute-envs/aws-cloud#required-permissions), and [Amazon EKS](../compute-envs/eks#required-platform-iam-permissions) compute environments. For Azure Blob Storage, see the [Azure Cloud data-links permissions](../compute-envs/azure-cloud#data-links). :::note When more than one workspace credential can access the same repository, Data Explorer lists the repository once and uses one of those credentials for every action on it: browsing, previewing, downloading, uploading, and deleting. Data Explorer does not select that credential in a deterministic order. If that credential lacks the permission an action needs, the action fails. Data Explorer does not retry with the other credentials. For example, if a read-only credential and a read-write credential both reach the same bucket, uploads can fail even though the read-write credential would succeed. To control which credential Data Explorer uses, add a data-link for the repository manually and select that credential from the **Credentials** drop-down. Use that data-link for actions that need the credential's permissions. ::: - **Configure individual data repositories manually** Select **Add data repository** from the Data Explorer tab to add a link to an individual repository (or prefix within a cloud bucket). Specify the **Provider**, **Path**, **Name**, **Credentials**, and **Description**, then select **Add**. For public cloud buckets, select **Public** from the **Credentials** drop-down. :::note Add a data-link at the root of any bucket or container used as a pipeline work directory, such as `s3://my-bucket`. Seqera Platform matches a run's work directory only against bucket-root data-links. A data-link scoped to a prefix such as `s3://my-bucket/work` does not open the work directory in Data Explorer. See [Isolate view, read, and write permissions to specific data repository paths](#isolate-view-read-and-write-permissions-to-specific-data-repository-paths). ::: ## Remove data repository links A workspace maintainer can remove a manually created data-link to a repository. From the **Data Explorer** tab, find the data repository that you want to remove. Select the options menu for the repository, and select **Remove**. When prompted, select **Remove** from the confirmation modal that appears. ## Browse data repositories ![](./_images/data_explorer.png) - **View data repository details** To view details such as the cloud provider, address, and credentials, select the information icon next to a data-link in the Data Explorer list. - **Search and filter data repositories** Search for repositories by name and region (for example, `region:eu-west-2`) in the search field, and filter by provider. - **Hide data repositories from list view** Using checkboxes, choose one or more data repositories, then select the **Hide** icon in the Data Explorer toolbar. To hide repositories individually, select **Hide** from the three dots options menu of a repository in the list. The Data Explorer list filter defaults to **Only visible**. Select **Only hidden** or **All** from the filtering menu to view hidden data repositories in the list. You can unhide a data repository by selecting **Show** from the three dots options menu in the list view. - **View data repository contents** Select a data-link from the Data Explorer list to view the contents of that data repository. From the **View data repository** page, you can browse directories and search for objects by name in a particular directory. The size and last-modified timestamp appear in columns to the right of the object name. You can also copy the path to the object to the clipboard or create a custom data-link (if the target is a directory). Any workspace role can download the object. Users with the Maintain role or above can also delete it. On the Data Explorer landing page you can view data repository details such as the provider, address, and credentials by selecting the information icon. You can also show or hide the data repository, or delete a custom data-link. - **Preview and download files** From the **View data repository** page, you can preview and download files. Select the download icon in the **Actions** column to download a file directly from the list view. Select a file to open a preview window that includes a **Download** button. File preview is supported for these object types: - Nextflow output files (`.command.*`, `.fusion.*`, and `.exitcode`) - Molecular data using the [Mol* library](https://molstar.org/) - Genome tracks using the [igv.js library](https://igv.org/doc/igvjs/) (annotations, wigs, alignments, and variants) - Text - CSV and TSV - PDF - HTML - Images (JPG, PNG, and SVG) :::note Except for genome tracks, the preview file size limit is 10 MB. You can still download files of 10-25 MB directly. Seqera Enterprise users can increase the default 25 MB download limit with `tower.content.max-file-size` in the `tower.yml` [configuration](https://docs.seqera.io/platform-enterprise/enterprise/configuration/overview#data-features) file. Increasing this value can degrade Platform performance. ::: :::note Data Explorer previews an HTML file in isolation, using a pre-signed URL scoped to that file only. Relative references to other files, such as hyperlinks to sibling report pages, images, stylesheets, and scripts, fail with an access denied error. To preview a multi-page report, generate it as a single self-contained HTML file that inlines its assets and uses JavaScript to show and hide sections. ::: - **Copy object paths** Select the **Path** of an object on the **View data repository** page to copy its absolute path to the clipboard. Use these object paths to specify input data locations during [pipeline launch](../launch/launchpad), add them to a [dataset](../data/datasets) for pipeline input, or when mounting data during Studio creation. ### Preview genome files with IGV Data Explorer renders genome tracks in the browser using the [igv.js library](https://igv.org/doc/igvjs/). Select a supported file, such as a BAM or BED file, from the **View data repository** page to open the viewer. You do not need to download the file or start a Studio. The viewer requests file data directly from your bucket. Apply a [CORS configuration](#cors-configurations-for-cloud-providers) to each bucket or storage account that holds genome files you want to preview. For the full IGV desktop application, create an [Xpra Studio with IGV](../getting-started/studios#xpra-visualize-genetic-variants-with-igv) instead. ### View lineage data for objects When a Nextflow run with data lineage enabled produced the object, the top of the object preview shows its lineage data: | Field | Description | |-------|-------------| | **Produced by** | The run and task that produced the object. Select the run or the task to open it. | | **Source for** | The tasks that used the object as an input. Select a task to open it. | | **Lineage labels** | The labels assigned to the object when the run published it. | If the object has no lineage data, no lineage fields appear in the preview. To capture lineage data for a workspace's runs, see [Lineage](../orgs-and-teams/workspace-management#lineage). ### Isolate view, read, and write permissions to specific data repository paths To isolate pipeline or Studios view, read, and write permissions to a specific **data repository path**, workspace maintainers can create **custom data-links**. A custom data-link is an individual data repository plus a manually configured path to a specific folder or directory. You can create one at any level of the data repository path hierarchy, provided the path is a folder (also known as a **prefix**). You can **Hide** or **Show** either the base data repository or any related custom data-links in Data Explorer using the **Show/Hide** toggle and the **Show data repositories** filter options: - Only visible (default) - Only hidden - All :::note This customized Data Explorer view displays by default for all workspace users until a workspace maintainer updates or removes the filter. ::: Custom data-links scoped to a prefix do not resolve run work directories. Seqera Platform matches a run's work directory to a data-link whose path is the root of the bucket or container, such as `s3://my-bucket` rather than `s3://my-bucket/work`. It strips the path below the root before matching, searches visible data-links only, and reapplies the path once a data-link matches. You can still browse a data-link registered at `s3://my-bucket/work`, but it never matches a run whose work directory is `s3://my-bucket/work/a1/b2c3d4`. A hidden bucket-root data-link has the same effect as no bucket-root data-link. To keep the work directory view, add a visible data-link at the root of each bucket or container used as a pipeline work directory. Your existing prefix-scoped data-links continue to work unchanged. For the symptoms and resolution, see [Work directory cannot be viewed in Data Explorer](../troubleshooting_and_faqs/data_explorer_troubleshooting#work-directory-cannot-be-viewed-in-data-explorer). :::warning A visible bucket-root data-link lets every workspace member, including participants with the View role, browse and download the whole bucket. This removes the path isolation that your prefix-scoped data-links provide. Hiding the bucket-root data-link does not narrow this access, because hidden data-links remain reachable. The credentials attached to a data-link define what Data Explorer reaches through it. See [Access control](#access-control). ::: ## Upload files to private data repositories Data Explorer supports single or bulk file uploads to your private data repositories. From the **View data repositories** page, select **Upload** and choose either the **Upload files** or **Upload folder** option. You can also drag and drop files and folders directly into Data Explorer. You can upload up to 300 files at a time via the Platform interface. The file size upload limits reflect the size limitations of the relevant cloud storage provider or data repository integration. These limits apply to cloud providers: - [AWS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html) - Single `PUT` upload: 5 GiB - Multi-part upload: 5 TiB - [Azure](https://learn.microsoft.com/en-us/rest/api/storageservices/put-blob?tabs=microsoft-entra-id#remarks) - Single `PUT` upload: 5 GiB - Multi-part upload: 4.77 TiB - [Cloudflare R2](https://developers.cloudflare.com/r2/platform/limits/) - Single `PUT` upload: 4.995 GiB - Multi-part upload: 50 TiB - [GCP](https://cloud.google.com/storage/quotas#objects): - Single `PUT` upload: 5 TiB - Multi-part upload: 5 TiB - [MinIO](https://docs.min.io/enterprise/aistor-object-store/reference/aistor-server/thresholds/) - Single `PUT` upload: 5 TiB - Multi-part upload: 50 TiB - [Oracle Cloud](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/managingobjects_topic-To_upload_objects_to_a_bucket.htm) - Single `PUT` upload: 64 MiB - Multi-part upload: 50 GiB To cancel an upload, select **X** in the upload window. Any files not uploaded display as **Failed**. Files that uploaded successfully are not removed. :::note You must configure cross-origin resource sharing (CORS) for your data repository provider to allow file uploads from Platform. CORS configuration differs for each provider. ::: ## Download multiple files You can download up to 1,000 files using the browser interface, or an unlimited number of files with the auto-generated download script that uses your data repository provider's CLI and credentials. :::note If you use a non-Chromium based browser, such as Safari or Firefox, file paths are concatenated with an underscore (`_`) character and the data repository directory structure is not reproduced locally. For example, the file `s3://example-us-east-1/path/to/files/my-file-1.txt` is saved as `path_to_files_my-file-1.txt`. ::: Open the data repository and navigate to the folder that you want to download files and folders from. By default, you can download the contents of the current directory by selecting **Download current directory**. Alternatively, use checkboxes to select specific files and folders, and select the **Download** button. You can **Download files** via the browser or **Download using code**. The code snippet is specific to the data repository provider you configured. You may be prompted to authenticate during the download process. Refer to your data repository provider's documentation for troubleshooting credential-related issues: - [GCP](https://cloud.google.com/sdk/gcloud/reference/storage) - [AWS](https://docs.aws.amazon.com/cli/latest/reference/s3/) - [Azure](https://learn.microsoft.com/en-us/cli/azure/storage?view=azure-cli-latest) ## CORS configurations for cloud providers Each cloud provider has a specific way to allow Cross-Origin Resource Sharing (CORS) for uploads, multi-file downloads, and genome file previews (IGV). ### Amazon S3 CORS configuration Apply a [CORS configuration](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ManageCorsUsing.html) to enable file uploads, folder downloads, and genome file previews (IGV) from Seqera Platform to and from specific S3 buckets. The CORS configuration is a JSON file that defines the origins, headers, and methods allowed for resource sharing requests to a bucket. Follow [these AWS instructions](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enabling-cors-examples.html) to apply the following CORS configuration to each bucket you want to enable file uploads, folder downloads, and genome file previews for: **Seqera Cloud S3 CORS configuration** ```json [ { "AllowedHeaders": ["*"], "AllowedMethods": ["PUT", "POST", "DELETE", "GET"], "AllowedOrigins": ["https://cloud.seqera.io"], "ExposeHeaders": ["ETag"] } ] ``` **Seqera Enterprise S3 CORS configuration** Replace `` with your Seqera Enterprise server URL: ```json [ { "AllowedHeaders": ["*"], "AllowedMethods": ["PUT", "POST", "DELETE", "GET"], "AllowedOrigins": ["https://"], "ExposeHeaders": ["ETag"] } ] ``` ### Azure Blob Storage CORS configuration :::note CORS configuration in Azure Blob Storage is set at the account level. This means that CORS rules for your account apply to every blob in the account. ::: Apply a [CORS configuration](https://learn.microsoft.com/en-us/rest/api/storageservices/cross-origin-resource-sharing--cors--support-for-the-azure-storage-services#enabling-cors-for-azure-storage) to enable file uploads, folder downloads, and genome file previews (IGV) from the Seqera Platform to and from your Azure Blob Storage account. **Seqera Cloud Azure CORS configuration** 1. From the [Azure portal](https://portal.azure.com), go to the **Storage account** you want to configure. 2. Under **Settings** in the left navigation menu, select **Resource sharing (CORS)**. 3. Add a new entry under **Blob service**: - **Allowed origins**: `https://cloud.seqera.io` - **Allowed methods**: `GET,POST,PUT,DELETE,HEAD` - **Allowed headers**: `x-ms-blob-type,content-type` - **Exposed headers**: `x-ms-blob-type` 4. Select **Save** to apply the CORS configuration. **Seqera Enterprise Azure CORS configuration** 1. From the [Azure portal](https://portal.azure.com), go to the **Storage account** you want to configure. 2. Under **Settings** in the left navigation menu, select **Resource sharing (CORS)**. 3. Add a new entry under **Blob service**: - **Allowed origins**: `https://` - **Allowed methods**: `GET,POST,PUT,DELETE,HEAD` - **Allowed headers**: `x-ms-blob-type,content-type` - **Exposed headers**: `x-ms-blob-type` 4. Select **Save** to apply the CORS configuration. ### Google Cloud Storage CORS configuration Apply a [CORS configuration](https://cloud.google.com/storage/docs/cross-origin#cors-components) to enable file uploads and genome file previews (IGV) from Seqera to specific GCS buckets. The CORS configuration is a JSON file that defines the origins, headers, and methods allowed for resource sharing requests to a bucket. Follow [these Google instructions](https://cloud.google.com/storage/docs/using-cors#command-line) to apply the following CORS configuration to each bucket you want to enable file uploads and genome file previews for. :::note Google Cloud Storage supports CORS configuration only through the gcloud CLI. ::: **Seqera Cloud GCS CORS configuration** ```json { "origin": ["https://cloud.seqera.io"], "method": ["GET", "POST", "PUT", "DELETE", "HEAD"], "responseHeader": ["Content-Type", "Content-Range"], "maxAgeSeconds": 3600 } ``` **Seqera Enterprise GCS CORS configuration** ```json { "origin": ["https://"], "method": ["GET", "POST", "PUT", "DELETE", "HEAD"], "responseHeader": ["Content-Type", "Content-Range"], "maxAgeSeconds": 3600 } ``` [roles]: ../orgs-and-teams/roles [wif]: ../credentials/workload_identity [wif-audit]: ../credentials/workload_identity#cloud-audit-attribution --- ## Data lineage :::info Data lineage in Platform is in public preview. It is supported in AWS compute environments. It requires Nextflow v25.04 or later, AWS S3 object storage, and Amazon Simple Notification Service (SNS). ::: :::warning The feature is experimental and subject to change. ::: Data lineage tracks the full provenance of every pipeline run at both the task and workflow level, including what executed, what data it consumed, and what outputs it produced. Use it to audit results, verify reproducibility, and trace file provenance. ## Why use data lineage Production pipelines generate results that teams need to trust, audit, and reproduce. Data lineage provides a precise, immutable record of how each result was produced. - **Reproducibility**: Every run, task, and output file receives a unique lineage ID (LID), a traversable URI that points to a structured record of what ran. Verify that two runs produced identical results, or identify where they diverged. - **Auditing and compliance**: For teams in regulated industries such as pharma, clinical genomics, and contract research organizations (CROs), lineage provides the audit trail needed for regulatory compliance. Each record captures inputs, outputs, parameters, compute environment, and the user who launched the run. - **Debugging**: When a cached task re-executes, or a pipeline produces an unexpected result, lineage traces backward from any output to all contributing tasks and parameters. Compare two task runs to isolate what changed. - **Broader team access**: Exploring Nextflow lineage previously required CLI access and the ability to read raw JSON. Platform now shows lineage data on pipeline run detail pages and in Data Explorer. - **Pipeline output visibility**: When lineage is enabled and a pipeline uses the [Nextflow workflow output syntax][nextflow-workflow-outputs] (Nextflow 24.10.0 or later), all published output files appear in the **Pipeline outputs** sub-tab on the [run details page][run-details]. Each file entry includes its lineage ID, lineage labels, and a direct link to Data Explorer, so any team member can locate and open a result without navigating cloud storage. - **Discovery across runs**: [Workflow output labels][workflow-labels] make output files discoverable across runs. Navigate lineage records by label to find all matching outputs workspace-wide, without knowing which specific run produced a file. ## Lineage records and event delivery Nextflow creates a structured JSON record for each entity in your pipeline when lineage is enabled: | Record type | Description | |---|---| | **WorkflowRun** | Full pipeline execution: repository, commit ID, parameters, compute environment, session ID, and Platform context (user, workspace, pipeline) | | **TaskRun** | Individual task execution: script, code checksum, inputs, outputs, container, and dependencies | | **FileOutput** | Output file: path, checksum, size, timestamp, and links back to the task and workflow that produced it | Each record gets a lineage ID (LID), a `lid://` URI that uniquely identifies the entity. ### Functional flow 1. Nextflow appends lineage record objects (`*.data.json`) to the configured object storage bucket. 1. The bucket filters for objects matching `.data.json` and publishes `s3:ObjectCreated:*` events to an SNS topic. 1. The SNS topic pushes each event to a per-workspace Platform webhook over HTTPS. 1. Platform verifies each delivery, buffers it, then reads the lineage object from the bucket and indexes it in the database. 1. The index enriches the [run details][run-details] and the display of workflow-generated objects in Data Explorer. :::info Because delivery is a push over HTTPS, `TOWER_SERVER_URL` must resolve to an **HTTPS** endpoint that AWS can reach from the public internet. SNS refuses plain HTTP and cannot resolve a private address. An installation AWS cannot reach receives no lineage events. Nextflow still writes the records to your bucket, and Platform indexes them once delivery is established. ::: :::note Lineage event ingestion moved from polling an Amazon SQS queue to SNS notifications pushed to Platform. If you are upgrading an installation that already has lineage configured, see [Data lineage event ingestion moves from SQS to SNS](../enterprise/upgrade#data-lineage-event-ingestion-moves-from-sqs-to-sns) for the permissions needed during the upgrade. ::: ## Enable data lineage To start collecting data lineage for all pipeline runs in your workspace: 1. Open **Settings > Workspace settings**. 2. Select **Lineage**. If you don't see **Lineage** listed, contact your system administrator. 3. Toggle **Enable lineage by default** on to collect data lineage for all pipeline runs in the workspace, or off to require per-pipeline launch configuration. Choose either a **Manual** or an **Automatic** configuration for lineage resources: - **Manual**: Use your own pre-provisioned bucket and SNS topic. Define the credentials, region, bucket name, and SNS topic ARN. After saving, subscribe the webhook URL shown on the settings page to your topic. See [Configure lineage manually](#configure-lineage-manually). - **Automatic**: Define the credentials and region. Platform creates the bucket, the SNS topic, the topic policies, the webhook subscription, and the bucket notification rule. This is the default setting. 4. Once set and enabled, all pipeline runs in the workspace generate data lineage. See [Lineage][workspace-lineage] for more information about the settings. :::note Lineage credentials must be key-based or role-based AWS credentials. Lineage does not support [workload identity federation](../credentials/workload_identity#limitations) credentials. ::: :::danger Updating the lineage settings after pipelines have generated lineage data results in historical data loss. The lineage index is tied to the lineage storage bucket. Changing it makes existing records inaccessible. To avoid data loss when updating the storage location, first copy all existing lineage data to the root of the new bucket (for example, `aws s3 sync s3://old-lineage-bucket s3://new-lineage-bucket`), then update the workspace setting. ::: When launching a pipeline in a data-lineage enabled workspace, the **Enable lineage** toggle in the pipeline **Run setup** reflects the **Enable lineage by default** workspace setting. Turn it off to _explicitly exclude_ data lineage for the pipeline run. :::tip Users with the Maintain role or above can toggle lineage on or off when launching a specific pipeline run. ::: ### Additional IAM permissions required If you use existing AWS Batch or AWS Cloud compute environments with custom IAM roles, the following service role policies are required: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "ListObjectsInBucket", "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": "arn:aws:s3:::seqera-lineage-" }, { "Sid": "AllObjectActions", "Effect": "Allow", "Action": "s3:*Object", "Resource": "arn:aws:s3:::seqera-lineage-/*" }, { "Sid": "AllowObjectTagging", "Effect": "Allow", "Action": [ "s3:PutObjectTagging", "s3:GetObjectTagging" ], "Resource": "arn:aws:s3:::seqera-lineage-/*" } ] } ``` Platform integration credentials require the following additional permissions for **Automatic** provisioning, which creates the bucket, the notification topic, and the webhook subscription: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "ManageNotificationTopics", "Effect": "Allow", "Action": [ "sns:CreateTopic", "sns:SetTopicAttributes", "sns:Subscribe", "sns:ConfirmSubscription", "sns:Unsubscribe", "sns:DeleteTopic" ], "Resource": "arn:aws:sns:*:*:seqera-lineage-*" }, { "Sid": "ManageLineageBuckets", "Effect": "Allow", "Action": [ "s3:CreateBucket", "s3:GetBucketNotification", "s3:PutBucketNotification", "s3:GetObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::seqera-lineage-*", "arn:aws:s3:::seqera-lineage-*/*" ] } ] } ``` No `sqs:*` permission is required. Platform holds no permission over messaging infrastructure in your account. :::note `sns:ConfirmSubscription` and `s3:ListBucket` are both required, and both fail quietly if omitted: - Without `sns:ConfirmSubscription`, provisioning completes and the workspace reports as configured, but **Event delivery** shows **Failed** and nothing is indexed. Platform completes the SNS handshake through the API. The permission is required even though you can also confirm a subscription by hand in a browser. - Without `s3:ListBucket`, rebuilding a workspace's lineage index from its bucket fails with `AccessDenied`. Reindexing pages the store with `ListObjectsV2`. ::: In **Manual** mode, Platform makes no control-plane calls other than confirming its own webhook subscription. The credentials need only: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "ReadLineageBucket", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::", "arn:aws:s3:::/*" ] }, { "Sid": "ConfirmLineageWebhook", "Effect": "Allow", "Action": [ "sns:ConfirmSubscription" ], "Resource": "arn:aws:sns:::" } ] } ``` ### Configure lineage manually In **Manual** mode you own the bucket, the topic, and the subscription. :::warning Lineage records are always stored at the root of the bucket. In **Bucket name**, enter only the bucket name (for example, `my-lineage-bucket`), without `s3://` or a path. Sub-paths such as `my-lineage-bucket/team-a` are not supported: the settings save without error, but Platform discards every event from the bucket and no lineage is recorded for your runs. To keep lineage data separate per workspace, use a separate bucket and SNS topic for each workspace. ::: Before saving the workspace settings: 1. Create the S3 bucket and the SNS topic. 1. Attach a topic access policy that allows the bucket to publish to the topic: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowBucketToPublishEvents", "Effect": "Allow", "Principal": { "Service": "s3.amazonaws.com" }, "Action": "sns:Publish", "Resource": "arn:aws:sns:::", "Condition": { "ArnEquals": { "aws:SourceArn": "arn:aws:s3:::" } } } ] } ``` 1. Configure a bucket notification rule that sends `s3:ObjectCreated:*` events for the `.data.json` suffix to the topic. Lineage records are written at the bucket root, so don't add a `prefix` filter: ```json { "TopicConfigurations": [ { "Id": "LineageRecordCreated", "TopicArn": "arn:aws:sns:::", "Events": ["s3:ObjectCreated:*"], "Filter": { "Key": { "FilterRules": [ { "Name": "suffix", "Value": ".data.json" } ] } } } ] } ``` 1. Grant the compute environment's IAM role read/write access to the bucket, using the service role policy shown earlier. Grant access to the whole bucket (`arn:aws:s3:::` and `arn:aws:s3:::/*`), not to a prefix within it, and don't restrict `s3:ListBucket` with an `s3:prefix` condition. Nextflow checks that the bucket root exists before it writes lineage records, and a prefix-scoped grant can make the run fail with `Creating a bucket is not supported`. Then save the workspace lineage settings, copy the **Webhook URL** shown on the settings page, and subscribe it to your topic: ```bash aws sns subscribe \ --topic-arn arn:aws:sns::: \ --protocol https \ --notification-endpoint '' ``` SNS immediately posts a subscription confirmation to the endpoint. Platform verifies and confirms it with the workspace's lineage credentials. The **Event delivery** badge on the settings page moves from **Awaiting confirmation** to **Active**. :::tip Set a delivery policy on your topic or subscription to widen the retry schedule. The AWS default of three attempts over roughly a minute drops events across an ordinary Platform restart. Automatically provisioned topics use a wider schedule for this reason. Tune it with `TOWER_LINEAGE_SNS_MAX_RETRIES` and `TOWER_LINEAGE_SNS_MAX_DELAY_SECONDS`. ::: :::note The `.data.json` suffix filter is recommended to reduce cost and delivery volume, but it is not required. Platform discards any event whose object key does not end in `.data.json`. ::: ### Event delivery status After you save the settings, the lineage settings page reports the **Event delivery** status (**Active**, **Awaiting confirmation**, **Failed**, or **Not configured**) alongside the workspace's **Webhook URL**. Delivery status is independent of the configuration status. A workspace can be configured and writable while Platform receives nothing. If delivery does not become **Active**, confirm that AWS can reach the installation over public HTTPS and that the lineage credentials grant `sns:ConfirmSubscription`. Records already written to the bucket are intact, and Platform re-indexes them once delivery resumes. ### Test lineage for a single pipeline or run To test or troubleshoot data lineage for a _specific pipeline_, add the following to your **Nextflow config file** under **Advanced options** when _adding_ a pipeline to the Launchpad. ```groovy lineage.enabled = true lineage.store.location = '' ``` To test for a _single pipeline run_, add the same code to your **Nextflow config file** under **Advanced options** when _launching_ the pipeline run. :::warning If data lineage is defined for a workspace, only that data is displayed in Platform. Any unique _specific pipeline_ or _single pipeline run_ lineage data is only accessible via the AWS S3 console and other related services (such as Amazon Athena). ::: ## Lineage in the Platform UI Platform shows lineage data on the run details page and in Data Explorer. ### Workflow run details For a run executed with lineage enabled, the [run details page][run-details] displays lineage data across the following tabs: - **Run Info**: Shows the lineage ID, lineage labels, and the full Platform context captured at execution time, including user, workspace, compute environment, pipeline name, revision, and commit ID. - **Tasks**: Displays the lineage ID and lineage labels for each `TaskRun` alongside existing task data. You can trace any task back to its lineage record. The tab also displays all task file inputs and outputs, and the upstream and downstream tasks linked by lineage records. - **Inputs**: Lists all input datasets and parameters with file paths, types, and lineage IDs and lineage labels where available. - **Outputs**: Lists all `FileOutput` records linked to the workflow run, including output name, file path, type, lineage ID, and lineage labels. Files link directly to [Data Explorer][data-explorer]. ### Data Explorer Output objects from a lineage-enabled run display their LID and any lineage labels when you preview the object in Data Explorer. You can trace any file back to the pipeline run that produced it. ## Search data lineage records Global search is on by default. To hide the search bar, set `TOWER_GLOBAL_SEARCH_ENABLED=false`. See [Configuration overview](../enterprise/configuration/overview). Use the search bar in the top navigation to find workflow runs, tasks, pipelines, and output files across every workspace you can access. To open it, select **Search** or press `Cmd+K` (macOS) or `Ctrl+K` (Windows and Linux). Search covers only workspaces that have data lineage enabled and in which you are a participant. Results include only records you have permission to view. Results are ordered by most recently indexed and load as you scroll. An empty query returns the most recent records across all accessible workspaces. As you type, the field suggests keywords and, where supported, values. ### Search syntax A query is a series of space-separated tokens. Each token is either a `qualifier:value` pair or free text. Three rules apply to every qualifier: - A space between tokens is **AND**: `type:file label:qc` returns output files that carry the `qc` label. - A comma inside a value is **OR**: `type:workflow,task` returns workflow runs and tasks. - Repeating a qualifier is **AND**: `label:qc label:validated` returns records carrying both labels. Qualifier names and free text are case-insensitive. Free text matches any substring of the record value. For example, `salmon` matches any record whose value contains `salmon`. :::caution A record has exactly one type and lives in exactly one workspace. Repeating `type:` or `workspace:` returns an empty list because no record can match both values. For example, `type:workflow type:file` requires a record to be both a workflow run and a file. Use the comma form `type:workflow,file` to match either type. ::: ### Qualifiers | Qualifier | Accepts | Description | | --- | --- | --- | | `type:` | `workflow`, `task`, `file` | Restrict results to a record type. Also accepts the internal names `WorkflowRun`, `TaskRun`, and `FileOutput`. | | `label:` | Any label | Records tagged with the label. Covers both Platform labels and Nextflow lineage labels. | | `workspace:` | `organization/workspace` | Scope the search to one or more workspaces by fully qualified name. | | `workspaceId:` | Numeric workspace ID | Numeric alias for `workspace:`. | | `workflow:` | A `WorkflowRun` LID | Scope the search to a single run. Results include the run itself, its tasks, and its published output files. | | `pipeline:` | A pipeline name | Scope the search to a pipeline name. Results include the pipeline itself and the output files in its work directory. | | `task:` | A `TaskRun` LID | Scope the search to a single task. Results include the task itself and the output files in its work directory. | | Free text | Any string | Case-insensitive substring match on the record value. | The field suggests `workspace:`, `type:`, and `label:` as you type. Enter the remaining qualifiers manually. Selecting a suggested `type:` or `workspace:` value replaces the current value for that qualifier. Selecting a suggested `label:` value adds another `label:` term to the query. `workspace:` and `workspaceId:` set the scope of a search rather than filter its results. A query that contains only a workspace still returns that workspace's most recent records. Omit both to search every workspace available to you. Referencing a workspace you do not participate in returns an error rather than an empty list. :::caution Renaming pipelines after execution can cause data lineage consistency issues. Pipeline names are **mutable** by design (can be edited). Data lineage records are **immutable**. If you run a pipeline, generate data lineage records, and then rename the pipeline, the indexed data lineage records are not associated with the new pipeline name. ::: ### Examples | Query | Returns | | --- | --- | | `type:workflow,task` | Workflow run or task records | | `label:qc,validated` | Records labeled `qc` or `validated` | | `label:qc label:validated` | Records labeled both `qc` and `validated` | | `label:qc,draft label:validated` | Records labeled `validated` and either `qc` or `draft` | | `type:file salmon` | Output files whose value contains `salmon` | | `type:file multiqc pipeline:rnaseq` | Output files whose value contains `multiqc` and associated with the `rnaseq` pipeline | | `workspace:acme/dev label:qc` | Records labeled `qc` in the `acme/dev` workspace | | `workspace:acme/dev,acme/prod` | Records in the `acme/dev` or `acme/prod` workspace | | `workspace:acme/dev workspace:acme/prod` | Nothing, because a record lives in one workspace. Use the comma form instead. | | `workflow:lid://abc123` | The run `lid://abc123`, its tasks, and its published output files | | `workflow:lid://abc123 type:task` | The tasks of run `lid://abc123` | | `task:lid://abc123 type:file` | The output files of task `lid://abc123` | :::tip Lineage search is also available through the Platform API. The `GET /lineage/search` endpoint accepts the same query syntax in its `q` parameter and returns paginated results. See the [Platform API reference][platform-api] for the full set of lineage endpoints. ::: ## Lineage labels Assign lineage labels to output files using the `label` directive in your Nextflow process definitions. Both Seqera Platform labels and Nextflow lineage labels propagate to lineage records. Seqera Platform excludes resource labels because they relate to underlying compute resources, not the data itself. :::info Nextflow sets lineage labels at execution time, and you cannot change them. Seqera Platform labels are mutable. Updating Platform labels after a run completes can produce a mismatch between Platform run labels and lineage labels. This is expected behavior. ::: {/* links */} [workflow-labels]: https://docs.seqera.io/nextflow/workflow#labels [nextflow-workflow-outputs]: https://docs.seqera.io/nextflow/workflow#outputs [workspace-lineage]: ../orgs-and-teams/workspace-management#lineage [run-details]: ../monitoring/run-details [data-explorer]: data-explorer [platform-api]: https://docs.seqera.io/platform-api --- ## Datasets :::note This feature is only available in organization workspaces. ::: Datasets are CSV (comma-separated values) and TSV (tab-separated values) files stored in, or linked to, a workspace. Use them as pipeline inputs to simplify data management, reduce data-entry errors, and support reproducible analyses. On the datasets screen, you can: - Upload directly or link to an externally hosted dataset. - View the count of pipeline runs in the workspace that have used a specific dataset input. - Apply multiple labels to datasets for easier searching and grouping. - Sort datasets by name, most recently updated, and most recently used. - Hide datasets that are not used in the workspace. - View dataset metadata (created by, last updated, last used). - Edit dataset details (name, description, and labels). - Create new versions of an uploaded dataset. ## Benefits - Datasets reduce errors from manual data entry when you launch pipelines. - Datasets can be generated automatically in response to events (such as new-file notifications from S3 storage). - Datasets can simplify differential data analysis when you use the same pipeline to launch a run for each dataset as it becomes available. ## Format The most commonly used datasets for Nextflow pipelines are sample sheets, where each row contains a sample identifier, the location of that sample's files (such as FASTQ files), and other sample details. For example, [*nf-core/rnaseq*](https://github.com/nf-core/rnaseq) works with input datasets (sample sheets) that include sample names, FASTQ file locations, and strandedness annotations. The Seqera Community Showcase sample dataset for *nf-core/rnaseq* looks like this: **Example rnaseq dataset** |sample |fastq_1 |fastq_2 |strandedness| |-------------------|------------------------------------|---------------------------------------------|------------| |WT_REP1 |s3://nf-core-awsmegatests/rnaseq/...|s3://nf-core-awsmegatests/rnaseq/... |reverse | |WT_REP1 |s3://nf-core-awsmegatests/rnaseq/...|s3://nf-core-awsmegatests/rnaseq/... |reverse | |WT_REP2 |s3://nf-core-awsmegatests/rnaseq/...|s3://nf-core-awsmegatests/rnaseq/... |reverse | |RAP1_UNINDUCED_REP1|s3://nf-core-awsmegatests/rnaseq/...| |reverse | |RAP1_UNINDUCED_REP2|s3://nf-core-awsmegatests/rnaseq/...| |reverse | |RAP1_UNINDUCED_REP2|s3://nf-core-awsmegatests/rnaseq/...| |reverse | |RAP1_IAA_30M_REP1 |s3://nf-core-awsmegatests/rnaseq/...|s3://nf-core-awsmegatests/rnaseq/... |reverse | :::note Use [Data Explorer](../data/data-explorer) to browse for cloud storage objects directly and copy the object paths to be used in your datasets. ::: ### Automation and pipeline schemas Combine datasets, [secrets](../secrets/overview), and [actions](../pipeline-actions/overview) to automate workflows that curate your data and maintain and launch pipelines in response to specific events. See [workflow-automation](https://seqera.io/blog/workflow-automation/) for an example of pipeline workflow automation. For your pipeline to use your dataset as input during runtime, information about the dataset and file format must be included in the relevant parameters of your [pipeline schema](../pipeline-schema/overview). The pipeline schema specifies the accepted dataset file type in the `mimetype` attribute (either `text/csv` or `text/tsv`). ## Dataset file content requirements and validation Datasets can point to files stored in Amazon S3, GitHub, Hugging Face, and other locations. To stage the file paths defined in the dataset, Nextflow requires access to the infrastructure where the files reside, whether on cloud or HPC systems. Add the access keys for data sources that require authentication to your [secrets](../secrets/overview). :::note Seqera doesn't validate your dataset file contents. While datasets can contain static file links, you're responsible for maintaining the access to that data. ::: ## Add a dataset All Seqera user roles have access to the datasets feature in organization workspaces. There are two ways to add a dataset: 1. **Direct upload**: Best when you need immutability and the file is under 10 MB. 2. **Link to an externally hosted file**: Best for large files, but availability and immutability depend on the external hosting service. ### Direct upload 1. In the sidebar navigation, select **Datasets**. 2. Select **Add Dataset** and choose **Upload file**. 3. Complete the **Name** and **Description** fields using information relevant to your dataset. 4. Optionally add one or more **Labels** to your dataset. You can use labels as a search filter but they don't apply to other resources in Seqera. 5. Upload a dataset to your workspace with drag-and-drop or use the **Upload file** file explorer dialog. 6. For datasets that use their first row for column names, customize the dataset view using the **Set first row as header** option. 7. Select **Add**. :::warning The size of the uploaded dataset file cannot exceed 10 MB. ::: ### Link to an externally hosted file 1. In the sidebar navigation, select **Datasets**. 2. Select **Add Dataset** and choose **Link to URL**. 3. Complete the **Name** and **Description** fields using information relevant to your dataset. 4. Optionally add one or more **Labels** to your dataset. You can use labels as a search filter but they don't apply to other resources in Seqera. 5. Copy and paste the dataset URL into the **Dataset URL** field. 6. For datasets that use their first row for column names, customize the dataset view using the **Set first row as header** option. 7. Select **Add**. 8. The dataset appears with a `Linked` badge. ## Manage dataset versions For directly uploaded datasets, Seqera can manage multiple versions. :::note For linked datasets, versioning is unavailable. ::: ### Add a dataset version 1. Select the three dots next to the dataset you want to add a new version for. 2. Select **Add version**. 3. Upload a dataset to your workspace with drag-and-drop or use the system **Upload file** file explorer dialog. 4. For datasets that use their first row for column names, customize the dataset view using the **Set first row as header** option. 5. Select **Add**. :::caution All subsequent versions of a dataset must be the same format (CSV or TSV) as the initial version. ::: ### View dataset versions To see all versions of a dataset, use the **Show** drop-down in the **Preview** tab. Seqera automatically displays a preview of the most recent version and flags it as **(latest)**, unless it is disabled. To preview previous dataset versions, change the version from the **Show** drop-down. The **Created by** and **Created on** values also change. To download a dataset version, select the **Download** icon. To copy a permalink to the dataset, select the **Copy** icon. ### Disable a dataset version To disable one or more dataset versions, select **Disable version**. You cannot select a disabled version as a pipeline input. If you disable the most recent version, the most recent non-disabled version is flagged as **(latest)**. :::note For compliance reasons, datasets or dataset versions cannot be deleted, they can only be **hidden** or **disabled**, respectively. Once disabled, a dataset version cannot be re-enabled. ::: ## Use a dataset To use a dataset with pipelines added to your workspace: 1. Open any pipeline that contains a pipeline schema from the [Launchpad](../launch/launchpad). 2. Select the input field for the pipeline, removing any default values. 3. Pick the dataset to use as input to your pipeline. :::note The input field drop-down displays only datasets that match the file type specified in the `nextflow_schema.json` of the chosen pipeline. If the schema specifies `"mimetype": "text/csv"`, no TSV datasets are available for use with that pipeline, and vice versa. If multiple dataset versions exist, the pipeline input always defaults to the **latest** version. ::: ## Manage datasets **View runs** To view a list of all pipeline runs in a workspace that have used a specific dataset input either: - Select the three dots next to a dataset and select **View runs**. - Select the number in the **Runs** column. **Toggle dataset visibility** Select the three dots next to a dataset and select **Mark dataset as hidden** to hide a dataset no longer used in your workspace. To show a hidden dataset, select **Mark dataset as visible**. This filter applies to all workspace users. You can toggle between **Visible**, **Hidden**, and **All** datasets in the **Show** drop-down on the main datasets page. :::note Hidden datasets do not count toward your per workspace quota. ::: **Filter datasets** Filter the list of datasets to only display datasets that match one or more filters defined in the **Search datasets** field. Select the info icon to see the list of available filters. **Edit dataset details** Select the three dots next to a dataset to edit the name, description, and labels associated with a dataset. --- ## Data privacy Seqera Platform orchestrates pipeline execution in your own infrastructure and stores only a limited set of metadata about your runs and tasks. ## Your data Your data stays within your infrastructure. To launch a pipeline with Seqera Platform, you create credentials and a compute environment in a workspace to connect your own infrastructure, such as high-performance computing (HPC) clusters, virtual machines (VMs), or Kubernetes. Seqera Platform uses this configuration to run the pipeline in your infrastructure, the same way the Nextflow CLI does. Seqera Platform does not manipulate your data, and your data is not transferred to the infrastructure where Seqera Platform runs. You can view some data in your storage from the Seqera Platform interface, such as logs and reports generated in a pipeline run. This data is never stored in Seqera Platform infrastructure. ## User deletion When a Seqera Platform user account is deleted: - The user account email is changed to `none@your-domain`. Runs and run metadata associated with the user account display that email address. - The username is changed to `username-`. - All of the user's organization, workspace, and team memberships are deleted. - All of the user's access tokens are deleted from their personal workspace. Enterprise installations also delete the following from the user's personal workspace: - All credentials - All compute environments - All actions created by the user ## Metadata stored by Seqera Platform The Nextflow runtime sends workflow execution metadata to Seqera Platform when: - You launch a pipeline from Seqera Platform. - You run a pipeline with the `-with-tower` command-line option. - You set `tower.enabled` in your Nextflow configuration. ### Workflow metadata Seqera Platform collects and stores the following metadata fields during a workflow execution: | Name | Description | | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `command_line` | The command line used to launch the workflow execution | | `commit_id` | The workflow project commit ID at the time of the execution | | `complete` | The workflow execution completion timestamp | | `config_files` | The Nextflow config file path(s) involved in the workflow execution | | `config_text` | The Nextflow config content used for the workflow execution. Note: secrets, such as AWS keys, are stripped and _not_ included in this field | | `container` | The container image name(s) used for the pipeline execution | | `container_engine` | The container engine name used for the pipeline execution | | `duration` | The workflow execution overall duration (wall time) | | `error_message` | The error message reported when the Nextflow execution fails | | `error_report` | The extended error message reported when the workflow execution fails | | `exit_status` | The workflow execution (POSIX) exit code | | `home_dir` | The launching user home directory path | | `launch_dir` | The workflow launching directory path | | `manifest_author` | The workflow project author as defined in the Nextflow config manifest file | | `manifest_default_branch` | The workflow project default Git branch as defined in the Nextflow config manifest file | | `manifest_description` | The workflow project description as defined in the Nextflow config manifest file | | `manifest_gitmodules` | The workflow project Git submodule flag in the Nextflow config manifest file | | `manifest_home_page` | The workflow project Git home page as defined in the Nextflow config manifest file | | `manifest_main_script` | The workflow project main script file name as defined in the Nextflow config manifest file | | `manifest_name` | The workflow project name as defined in the Nextflow config manifest file | | `manifest_nextflow_version` | The workflow project required Nextflow version defined in the Nextflow config manifest file | | `manifest_version` | The workflow project version string as defined in the Nextflow config manifest file | | `nextflow_build` | The build number of the Nextflow runtime used to launch the workflow execution | | `nextflow_timestamp` | The build timestamp of the Nextflow runtime used to launch the workflow execution | | `nextflow_version` | The version string of the Nextflow runtime used to launch the workflow execution | | `params` | The workflow params used to launch the pipeline execution | | `profile` | The workflow config profile string used for the pipeline execution | | `project_dir` | The directory path where the workflow scripts are stored | | `project_name` | The workflow project name | | `repository` | The workflow project repository | | `resume` | The flag set when a resume execution was submitted | | `revision` | The workflow project revision number | | `run_name` | The workflow run name as given by the Nextflow runtime | | `script_file` | The workflow script file path | | `script_id` | The workflow script checksum number | | `script_name` | The workflow script filename | | `session_id` | The workflow execution unique UUID as assigned by the Nextflow runtime | | `start` | The workflow execution start timestamp | | `stats_cached_count` | The number of cached tasks upon completion | | `stats_cached_duration` | The aggregate time of cached tasks upon completion | | `stats_cached_pct` | The percentage of cached tasks upon completion | | `stats_compute_time_fmt` | The overall compute time as a formatted string | | `stats_failed_count` | The number of failed tasks upon completion | | `stats_failed_count_fmt` | The number of failed tasks upon completion as a formatted string | | `stats_failed_duration` | The aggregate time of failed tasks upon completion | | `stats_failed_pct` | The percentage of failed tasks upon completion | | `stats_ignored_count` | The number of ignored tasks upon completion | | `stats_ignored_count_fmt` | The number of ignored tasks upon completion as a formatted string | | `stats_ignored_pct` | The percentage of ignored tasks upon completion | | `stats_succeed_count` | The number of succeeded tasks upon completion | | `stats_succeed_count_fmt` | The number of succeeded tasks upon completion as a formatted string | | `stats_succeed_duration` | The aggregate time of succeeded tasks upon completion | | `stats_succeed_pct` | The percentage of succeeded tasks upon completion | | `status` | The workflow execution status | | `submit` | The workflow execution submission timestamp | | `success` | The flag reporting whether the execution completed successfully | | `user_name` | The POSIX user name that launched the workflow execution | | `work_dir` | The workflow execution scratch directory path | ### Task metadata Seqera Platform collects and stores the following metadata fields for each task: | Name | Description | | -------------- | ---------------------------------------------------------------------------------------------- | | `attempt` | Number of Nextflow execution attempts of the task | | `cloud_zone` | Cloud zone where the task execution was allocated | | `complete` | Task execution completion timestamp | | `container` | Container image name used to execute the task | | `cost` | Estimated task compute cost | | `cpus` | Number of CPUs requested | | `disk` | Amount of disk storage requested | | `duration` | Amount of time for the task to complete | | `env` | Task execution environment variables | | `error_action` | Action applied on task failure | | `executor` | Executor requested for the task execution | | `exit_status` | Task POSIX exit code on completion | | `hash` | Task unique hash code | | `inv_ctxt` | Number of involuntary context switches | | `machine_type` | Cloud virtual machine type | | `memory` | Amount of memory requested | | `module` | Environment module requested | | `name` | Task unique name | | `native_id` | Task unique ID as assigned by the underlying execution platform | | `pcpu` | Percentage of CPU used to compute the task | | `peak_rss` | Peak of real memory during the task execution | | `peak_vmem` | Peak of virtual memory during the task execution | | `pmem` | Percentage of memory used to compute the task | | `price_model` | Cloud price model applied for the task | | `process` | Nextflow process name | | `queue` | Compute queue name requested | | `rchar` | Number of bytes the process read, using any read-like system call from files, pipes, and terminals | | `read_bytes` | Number of bytes the process directly read from disk | | `realtime` | Time required to compute the task | | `rss` | Real memory (resident set) size of the process | | `scratch` | Flag reporting the task was executed in a local scratch path | | `script` | Task command script | | `start` | Task execution start timestamp | | `status` | Task execution status | | `submit` | Task submission timestamp | | `syscr` | Number of read-like system call invocations that the process performed | | `syscw` | Number of write-like system call invocations that the process performed | | `tag` | Nextflow tag associated with the task execution | | `task_id` | Nextflow task ID | | `time` | Task execution timeout requested | | `vmem` | Virtual memory size used by the task execution | | `vol_ctxt` | Number of voluntary context switches | | `wchar` | Number of bytes the process wrote, using any write-like system call | | `workdir` | Task execution work directory | | `write_bytes` | Number of bytes the process wrote to disk | --- ## Source locations The images under this folder are both generated from source files in the Seqera company Google Drive. If you're a Seqera employee, search for the following in Google Drive for the respective source files: - "Seqera reference architecture" for the source file of 'seqera_reference_architecture.png'. - "References, Architectures & Diagrams" for the source file of 'seqera_reference_architecture_aws.png'. --- ## Cert_on_frontend title: "cert_on_frontend" This example assumes deployment on an Amazon Linux 2 AMI. 1. Install NGINX and other required packages: ```yml sudo amazon-linux-extras install nginx1.12 sudo wget -r --no-parent -A 'epel-release-*.rpm' https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/ sudo rpm -Uvh dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-*.rpm sudo yum-config-manager --enable epel* sudo yum repolist all sudo amazon-linux-extras install epel -y ``` 2. Generate a [private certificate and key](https://www.digitalocean.com/community/tutorials/openssl-essentials-working-with-ssl-certificates-private-keys-and-csrs). 3. Create a `ssl.conf` file. ```ini server { server_name your.server.name; # replace with your server name root /usr/share/nginx/html; location / { proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Authorization $http_authorization; proxy_pass_header Authorization; proxy_pass http://frontend/; proxy_read_timeout 90; proxy_redirect http://frontend/ https://your.redirect.url/; } error_page 404 /404.html; location = /40x.html { } error_page 500 502 503 504 /50x.html; location = /50x.html { } listen [::]:443 ssl ipv6only=on; listen 443 ssl; ssl_certificate /etc/ssl/testcrt.crt; ssl_certificate_key /etc/ssl/testkey.key; } ``` 4. Make a local copy of the `frontend` container's `/etc/nginx/nginx.conf` file. 5. Add the following to the `server` block of your local `nginx.conf` file: ```ini include /etc/nginx/ssl.conf; ``` 6. Modify the `frontend` container definition in your `docker-compose.yml` file: ```yml frontend: image: cr.seqera.io/frontend:${TAG} networks: - frontend ports: - 8000:80 - 443:443 volumes: - $PWD/nginx.conf:/etc/nginx/nginx.conf - $PWD/ssl.conf:/etc/nginx/ssl.conf - $PWD/cert/testcrt.crt:/etc/ssl/testcrt.crt - $PWD/cert/testkey.key:/etc/ssl/testkey.key restart: always depends_on: - backend ``` --- ## Custom Content Security Policy headers ## Introduction HTTP security headers are an important part of a website's security posture. They protect against different types of attacks including cross-site scripting (XSS), SQL injection, and clickjacking. Object storage is external to Seqera Platform, and read and write access is strictly limited to a selected group of object storage providers. These select providers are explicitly defined in the Content Security Policy (CSP). ## Supported object storage providers Data Explorer can read from, and write to, the following object storage providers by default: - [Amazon S3][aws-s3] - [Google Cloud Object Storage][gcp-os] - [Azure Blob Storage][azure-bs] - [OCI Object Storage][oci-os] - [Cloudflare R2][cloudflare-r2] - [LakeFS Cloud][lakefs] ## Subdomain support If your object storage provider and Seqera deployment share the same subdomain (e.g., `minio.janedoepharma.com` and `platform.janedoepharma.com`), then communication between Seqera and the provider works **without** additional customization. However, if your object storage provider and subdomain don't match, the CSP headers need to be customized. ## Connecting additional providers Accessing new object storage providers in [Data Explorer][data-explorer] requires updating the Content Security Policy to include the domains to access. This is done by setting the `ADDITIONAL_CSP` environment variable for the frontend container. :::note This configuration requires the [Seqera frontend](../platform-kubernetes#seqera-frontend-unprivileged) image, which is the only frontend image published from 26.2. On an earlier release still running the legacy root image, contact Seqera support. ::: ### Configuration Set the `ADDITIONAL_CSP` environment variable with a space-separated list of domains to add to the Content Security Policy. For example, to add support for MinIO: ```bash ADDITIONAL_CSP="https://*.min.io" ``` To add multiple domains: ```bash ADDITIONAL_CSP="https://*.min.io https://custom-storage.example.com" ``` :::info If your object storage is accessed on a port other than port **80**, include the port in the address (e.g., `https://myobjectstorage.min.io:9000`). ::: {/* links */} [aws-s3]: https://aws.amazon.com/s3/ [gcp-os]: https://cloud.google.com/storage [azure-bs]: https://azure.microsoft.com/en-us/products/storage/blobs [oci-os]: https://www.oracle.com/cloud/storage/object-storage/ [cloudflare-r2]: https://www.cloudflare.com/developer-platform/products/r2/ [lakefs]: https://lakefs.io/ [data-explorer]: ../../data/data-explorer.md --- ## Custom AWS Batch launch container You can customize your Seqera instance's Nextflow launch container, e.g., to include private CA certificates or compliance software in your Nextflow environment. :::caution Seqera recommends using the default Nextflow launch container wherever possible. Custom launch containers can complicate your Seqera configuration and upgrade process. ::: :::note A custom launch container determines the Nextflow runtime for every run. It takes precedence over per-run version selection. When `TOWER_LAUNCH_CONTAINER` is set, Seqera Platform hides the [**Nextflow version**](../../launch/advanced#nextflow-version) selector on all compute environments, and any selected version has no effect. ::: Specify the path to your custom launch container image with an environment variable: ```env TOWER_LAUNCH_CONTAINER=quay.io/seqeralabs/nf-launcher:j17-23.04.3 ``` **Use an AWS Batch job definition as a Seqera custom launch container** Seqera Platform automatically registers an AWS Batch [job definition](https://docs.aws.amazon.com/batch/latest/userguide/job_definitions.html) to launch pipelines with the required Nextflow runtime. If you need to manage this manually, create a job definition in your AWS Batch environment with the following settings: - `name`: any of your choice - `image`: a custom image based on the Seqera [nf-launcher image](https://quay.io/repository/seqeralabs/nf-launcher) - `vcpus`: at least `1` - `memory`: at least `1000` - `command`: `true` After the job definition is registered, update your Seqera Enterprise configuration with the following (replace `` with the name of the job definition): :::caution The custom launch container is set at the root level, so all executions in your Seqera instance will use this container. If you set an AWS Batch job definition as your custom launch container, launching workflow executions in other cloud provider compute environments will fail. ::: ```env TOWER_LAUNCH_CONTAINER=job-definition:// ``` :::note The repository where your launch container resides must be accessible to the Batch cluster's [ECS Agent](https://docs.aws.amazon.com/batch/latest/userguide/private-registry-auth.html). ::: --- ## Firewall configuration Seqera Platform Cloud ([cloud.seqera.io](https://cloud.seqera.io)) may need to connect to resources within your network, e.g., your storage server. To do so, your firewall must be configured to allow certain IPs to reach your resources. A dynamic list of IPs is kept up-to-date at [`meta.seqera.io`](https://meta.seqera.io). This endpoint returns a JSON object that can be parsed to dynamically adapt your firewall, e.g., in Python with the `requests` package: ```python $ python3 >>> import requests >>> requests.get("https://meta.seqera.io").json() { "egress": ["a.b.c.d/32", "..."], "ingress": ["e.f.g.h/32", "..."] } ``` The response contains two lists. The list you need depends on the direction of the traffic: | To allow | Use | The list contains | | :------- | :-- | :---------------- | | Seqera to reach a resource in your network, such as your storage server, container registry, or Git server | `egress` | The addresses Seqera connects from | | Your Platform Enterprise installation or a compute environment to reach a Seqera service, such as `wave.seqera.io` | `ingress` | The addresses Seqera services are reachable at | You only need the `ingress` list if your outbound rules are IP-based rather than DNS-based. For DNS-based rules, see [DNS allowlist](#dns-allowlist). :::note Always query [`meta.seqera.io`](https://meta.seqera.io) when you build or review your firewall rules. The addresses change as services are added, so a copied list becomes stale. ::: ### Wave Wave traffic runs in both directions, and each direction uses a different list: - To let Wave read from or write into your container registry, allow the `egress` addresses in your registry's network rules. Wave connects to your registry from Seqera's network in every mode, including augment, mirror, and freeze. - To let your installation or compute environment call Wave, allow the `ingress` addresses in your outbound rules if those rules are IP-based. Wave cannot reach a registry that is exposed only through a private endpoint inside your network, and allowlisting does not change this. See [Registries behind a private endpoint](../configuration/wave#registries-behind-a-private-endpoint). ### DNS allowlist In order for you to access resources such as Fusion tarballs, `nf-xpack` files, Wave cloud containers and other services provided by Seqera, you'll need to add `*.seqera.io.cdn.cloudflare.net` to the allowlist in your network firewall. If DNS wildcards aren't supported by your firewall, you can use the following: - `cloud.seqera.io` - `api.cloud.seqera.io` - `user-data.cloud.seqera.io` - `tower.nf` - `api.tower.nf` - `connect.cloud.seqera.io` and its subdomains `*.connect.cloud.seqera.io` - `hub.seqera.io` - `intern.seqera.io` - `ai.seqera.io` - `ai-api.seqera.io` - `wave.seqera.io` - `community.wave.seqera.io` - `cerbero.seqera.io` - `public.cr.seqera.io` - `auth.cr.seqera.io` - `cr.seqera.io` - `licenses.seqera.io` - `api.multiqc.info` - `fusionfs.seqera.io` - `nf-xpack.seqera.io` - `community-cr-prod.seqera.io` - `fusionfs.seqera.io` - `nf-xpack.seqera.io` - `public-cr-prod.seqera.io` - `wave-cache-prod-cloudflare.seqera.io` - `fusionfs.seqera.io.cdn.cloudflare.net` - `nf-xpack.seqera.io.cdn.cloudflare.net` - `community-cr-prod.seqera.io.cdn.cloudflare.net` - `fusionfs.seqera.io.cdn.cloudflare.net` - `nf-xpack.seqera.io.cdn.cloudflare.net` - `public-cr-prod.seqera.io.cdn.cloudflare.net` - `wave-cache-prod-cloudflare.seqera.io.cdn.cloudflare.net` - `registry.nextflow.io` (required from Nextflow 25.10) :::note Nextflow makes network calls to the `registry.nextflow.io` domain to resolve and download plugins. Even when plugins are already cached locally, Nextflow makes calls to the registry to check for updates and resolve plugin metadata. Ensure this domain is included in your allowlist to prevent pipeline execution failures. ::: If you chose to filter by specific DNS records, please note that new services may be added in the future. :::note If your allowlist is based on IP addresses, allow all of the following IP addresses: https://www.cloudflare.com/ips/. ::: --- ## Manual AWS Batch configuration This page describes how to set up AWS roles and Batch queues manually for the deployment of Nextflow workloads with Seqera Platform. :::tip Manual AWS Batch configuration is only necessary if you don't use Batch Forge. Batch Forge _automatically creates_ the AWS Batch queues required for your workflow executions. ::: Complete the following procedures to configure AWS Batch manually: 1. Create a user policy. 2. Create the instance role policy. 3. Create the AWS Batch service role. 4. Create an EC2 Instance role. 5. Create a Nextflow head job role. 6. Create an EC2 SpotFleet role. 7. Create a launch template. 8. Create the AWS Batch compute environments. 9. Create the AWS Batch queue. ### Create a user policy Create the policy for the user launching Nextflow jobs: 1. In the [IAM Console](https://console.aws.amazon.com/iam/home), select **Create policy** from the Policies page. 1. Create a new policy with the following content: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt1530313170000", "Effect": "Allow", "Action": [ "batch:CancelJob", "batch:RegisterJobDefinition", "batch:DescribeComputeEnvironments", "batch:DescribeJobDefinitions", "batch:DescribeJobQueues", "batch:DescribeJobs", "batch:ListJobs", "batch:SubmitJob", "batch:TerminateJob" ], "Resource": ["*"] } ] } ``` 1. Save with it the name `seqera-user`. ### Create the instance role policy Create the policy with a role that allows Seqera to submit Batch jobs on your EC2 instances: 1. In the [IAM Console](https://console.aws.amazon.com/iam/home), select **Create policy** from the Policies page. 1. Create a new policy with the following content: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "batch:DescribeJobQueues", "batch:CancelJob", "batch:SubmitJob", "batch:ListJobs", "batch:DescribeComputeEnvironments", "batch:TerminateJob", "batch:DescribeJobs", "batch:RegisterJobDefinition", "batch:DescribeJobDefinitions", "batch:TagResource", "ecs:DescribeTasks", "ec2:DescribeInstances", "ec2:DescribeInstanceTypes", "ec2:DescribeInstanceAttribute", "ecs:DescribeContainerInstances", "ec2:DescribeInstanceStatus", "logs:Describe*", "logs:Get*", "logs:List*", "logs:Create*", "logs:Put*", "logs:StartQuery", "logs:StopQuery", "logs:TestMetricFilter", "logs:FilterLogEvents" ], "Resource": "*" } ] } ``` 1. Save it with the name `seqera-batchjob`. ### Create the Batch Service role Create a service role used by AWS Batch to launch EC2 instances on your behalf: 1. In the [IAM Console](https://console.aws.amazon.com/iam/home), select **Create role** from the Roles page. 1. Select **AWS service** as the trusted entity type, and **Batch** as the service. 1. On the next page, the `AWSBatchServiceRole` is already attached. No further permissions are needed for this role. 1. Enter `seqera-servicerole` as the role name and add an optional description and tags if needed, then select **Create**. ### Create an EC2 instance role Create a role that controls which AWS resources the EC2 instances launched by AWS Batch can access: 1. In the [IAM Console](https://console.aws.amazon.com/iam/home), select **Create role** from the Roles page. 1. Select AWS service as the trusted entity type, EC2 as the service, and _EC2 - Allows EC2 instances to call AWS services on your behalf_ as the use case. 1. Select **Next: Permissions**. Search for the following policies to attach to the role: - `AmazonEC2ContainerServiceforEC2Role` - `AmazonS3FullAccess` (you may want to use a custom policy to allow access only on specific S3 buckets) - `seqera-batchjob` (the instance role policy created above) 1. Enter `seqera-instancerole` as the role name and add an optional description and tags if needed, then select **Create**. ### Create a Nextflow head job role Create an IAM role for the Nextflow head job. This role is attached to the Nextflow head job container and grants it the permissions needed to orchestrate workflow tasks and retrieve task logs from CloudWatch. You specify this role in the **Head Job role** field when creating a manual compute environment in Seqera Platform. :::note This role is separate from the EC2 instance role. The head job role is attached directly to the Nextflow container via the Batch job definition, while the instance role applies to the underlying EC2 instance. ::: 1. In the [IAM Console](https://console.aws.amazon.com/iam/home), select **Create policy** from the Policies page. 1. Create a new policy with the following content: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "BatchJobManagement", "Effect": "Allow", "Action": [ "batch:DescribeJobQueues", "batch:CancelJob", "batch:SubmitJob", "batch:ListJobs", "batch:DescribeComputeEnvironments", "batch:TerminateJob", "batch:DescribeJobs", "batch:RegisterJobDefinition", "batch:DescribeJobDefinitions", "batch:TagResource", "ecs:DescribeTasks", "ec2:DescribeInstances", "ec2:DescribeInstanceTypes", "ec2:DescribeInstanceAttribute", "ecs:DescribeContainerInstances", "ec2:DescribeInstanceStatus" ], "Resource": "*" }, { "Sid": "CloudWatchLogsAccess", "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:FilterLogEvents", "logs:GetLogEvents", "logs:ListTagsLogGroup", "logs:PutLogEvents", "logs:StartQuery", "logs:StopQuery", "logs:TestMetricFilter" ], "Resource": "*" } ] } ``` :::note `logs:GetLogEvents` is required for Nextflow to retrieve task stderr from CloudWatch when tasks fail. Without it, error reports for failed tasks show an `AccessDeniedException` instead of the actual task error. ::: 1. Save it with the name `seqera-headjob-policy`. 1. Select **Create role** from the Roles page. Select **AWS service** as the trusted entity type and **Elastic Container Service Task** as the use case. 1. Attach the `seqera-headjob-policy` policy to the role. 1. Enter `seqera-headjob-role` as the role name and select **Create**. ### Create an EC2 SpotFleet role The EC2 SpotFleet role allows you to use Spot instances when you run jobs in AWS Batch. Create a role for the creation and launch of Spot fleets — Spot instances with similar compute capabilities (i.e., vCPUs and RAM): 1. In the [IAM Console](https://console.aws.amazon.com/iam/home), select **Create role** from the Roles page. 1. Select AWS service as the trusted entity type, EC2 as the service, and _EC2 - Spot Fleet Tagging_ as the use case. 1. On the next page, the `AmazonEC2SpotFleetTaggingRole` is already attached. No further permissions are needed for this role. 1. Enter `seqera-fleetrole` as the role name and add an optional description and tags if needed, then select **Create**. ### Create a launch template Create a launch template to configure the EC2 instances deployed by Batch jobs: 1. In the [EC2 Console](https://console.aws.amazon.com/ec2/v2/home), select **Create launch template** from the Launch templates page. 1. Scroll down to **Advanced details** and paste the following in the **User data** field: ```bash MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="//" --// Content-Type: text/cloud-config; charset="us-ascii" #cloud-config write_files: - path: /root/custom-ce.sh permissions: 0744 owner: root content: | #!/usr/bin/env bash exec > >(tee /var/log/tower-forge.log|logger -t TowerForge -s 2>/dev/console) 2>&1 ## yum install -q -y jq sed wget unzip nvme-cli lvm2 ## install CloudWatch agent curl -s https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm -o amazon-cloudwatch-agent.rpm rpm -U ./amazon-cloudwatch-agent.rpm rm -f ./amazon-cloudwatch-agent.rpm curl -s https://nf-xpack.seqera.io/amazon-cloudwatch-agent/config-v0.4.json \ # | sed 's/$FORGE_ID//g' \ > /opt/aws/amazon-cloudwatch-agent/bin/config.json /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \ -a fetch-config \ -m ec2 \ -s \ -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json ## format NVMe storage for Fusion mkdir -p /scratch/fusion NVME_DISKS=($(nvme list | grep 'Amazon EC2 NVMe Instance Storage' | awk '{ print $1 }')) NUM_DISKS=${#NVME_DISKS[@]} if (( NUM_DISKS > 0 )); then if (( NUM_DISKS == 1 )); then mkfs -t xfs ${NVME_DISKS[0]} mount ${NVME_DISKS[0]} /scratch/fusion else pvcreate ${NVME_DISKS[@]} vgcreate scratch_fusion ${NVME_DISKS[@]} lvcreate -l 100%FREE -n volume scratch_fusion mkfs -t xfs /dev/mapper/scratch_fusion-volume mount /dev/mapper/scratch_fusion-volume /scratch/fusion fi fi chmod a+w /scratch/fusion ## ECS configuration mkdir -p /etc/ecs echo ECS_IMAGE_PULL_BEHAVIOR=once >> /etc/ecs/ecs.config echo ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE=true >> /etc/ecs/ecs.config echo ECS_ENABLE_SPOT_INSTANCE_DRAINING=true >> /etc/ecs/ecs.config echo ECS_CONTAINER_CREATE_TIMEOUT=10m >> /etc/ecs/ecs.config echo ECS_CONTAINER_START_TIMEOUT=10m >> /etc/ecs/ecs.config echo ECS_CONTAINER_STOP_TIMEOUT=10m >> /etc/ecs/ecs.config echo ECS_MANIFEST_PULL_TIMEOUT=10m >> /etc/ecs/ecs.config ## stop docker systemctl stop docker ## install AWS CLI curl -s https://nf-xpack.seqera.io/miniconda-awscli/miniconda-25.3.1-awscli-1.40.12.tar.gz \ | tar xz -C / export PATH=$PATH:/home/ec2-user/miniconda/bin ln -s /home/ec2-user/miniconda/bin/aws /usr/bin/aws ## restart docker systemctl start docker systemctl enable --now --no-block ecs ## kernel settings to prevent OOM echo "1258291200" > /proc/sys/vm/dirty_bytes echo "629145600" > /proc/sys/vm/dirty_background_bytes runcmd: - bash /root/custom-ce.sh --//-- ``` 1. To prepend a custom identifier to the CloudWatch log streams for AWS resources created by your manual compute environment, uncomment the `| sed 's/$FORGE_ID//g' \` line and replace `` with your custom identifier. If omitted, `$FORGE_ID` remains as-is in the config. 1. Save the template with the name `seqera-launchtemplate`. 1. In the [EC2 Console](https://console.aws.amazon.com/ec2/v2/home), select **Create launch template** from the Launch templates page. 1. Scroll down to **Advanced details** and paste the following in the **User data** field: ```bash MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="//" --// Content-Type: text/cloud-config; charset="us-ascii" #cloud-config write_files: - path: /root/custom-ce.sh permissions: 0744 owner: root content: | #!/usr/bin/env bash exec > >(tee /var/log/tower-forge.log|logger -t TowerForge -s 2>/dev/console) 2>&1 ## yum install -q -y jq sed wget unzip nvme-cli lvm2 ## install CloudWatch agent curl -s https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm -o amazon-cloudwatch-agent.rpm rpm -U ./amazon-cloudwatch-agent.rpm rm -f ./amazon-cloudwatch-agent.rpm curl -s https://nf-xpack.seqera.io/amazon-cloudwatch-agent/config-v0.4.json \ # | sed 's/$FORGE_ID//g' \ > /opt/aws/amazon-cloudwatch-agent/bin/config.json /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \ -a fetch-config \ -m ec2 \ -s \ -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json ## ECS configuration mkdir -p /etc/ecs echo ECS_IMAGE_PULL_BEHAVIOR=once >> /etc/ecs/ecs.config echo ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE=true >> /etc/ecs/ecs.config echo ECS_ENABLE_SPOT_INSTANCE_DRAINING=true >> /etc/ecs/ecs.config echo ECS_CONTAINER_CREATE_TIMEOUT=10m >> /etc/ecs/ecs.config echo ECS_CONTAINER_START_TIMEOUT=10m >> /etc/ecs/ecs.config echo ECS_CONTAINER_STOP_TIMEOUT=10m >> /etc/ecs/ecs.config echo ECS_MANIFEST_PULL_TIMEOUT=10m >> /etc/ecs/ecs.config ## stop docker systemctl stop docker ## install AWS CLI curl -s https://nf-xpack.seqera.io/miniconda-awscli/miniconda-25.3.1-awscli-1.40.12.tar.gz \ | tar xz -C / export PATH=$PATH:/home/ec2-user/miniconda/bin ln -s /home/ec2-user/miniconda/bin/aws /usr/bin/aws ## restart docker systemctl start docker systemctl enable --now --no-block ecs ## kernel settings to prevent OOM echo "1258291200" > /proc/sys/vm/dirty_bytes echo "629145600" > /proc/sys/vm/dirty_background_bytes runcmd: - bash /root/custom-ce.sh --//-- ``` 1. To prepend a custom identifier to the CloudWatch log streams for AWS resources created by your manual compute environment, uncomment the `| sed 's/$FORGE_ID//g' \` line and replace `` with your custom identifier. If omitted, `$FORGE_ID` remains as-is in the config. 1. Save the template with the name `seqera-launchtemplate`. ### Create the Batch compute environments :::caution AWS Graviton instances (ARM64 CPU architecture) are not supported in manual compute environments. To use Graviton instances, create your AWS Batch compute environment with [Batch Forge](../../compute-envs/aws-batch#automatic-configuration-of-batch-resources). ::: Nextflow makes use of two job queues during workflow execution: - A head queue to run the Nextflow application - A compute queue where Nextflow will submit job executions While the compute queue can use a compute environment with Spot instances, the head queue requires an on-demand compute environment. If you intend to use an on-demand compute environment for compute jobs, the same job queue can be used for both head and compute. :::note Spot instances can significantly reduce your AWS compute costs, provided your workflow compute tasks can run on ephemeral instances. ::: Create a compute environment for each queue in the AWS Batch console: The head queue requires an on-demand compute environment. Do not select **Use Spot instances** during compute environment creation. 1. In the [Batch Console](https://eu-west-1.console.aws.amazon.com/batch/home), select **Create** on the Compute environments page. 1. Select **Amazon EC2** as the compute environment configuration. :::note Seqera AWS Batch compute environments created with [Batch Forge](../../compute-envs/aws-batch#automatic-configuration-of-batch-resources) support using Fargate for the head job, but manual compute environments must use EC2. ::: 1. Enter a name of your choice, and apply the `seqera-servicerole` and `seqera-instancerole`. 1. When creating the Seqera compute environment, enter the ARN of `seqera-headjob-role` in the **Head Job role** field. 1. Enter vCPU limits and instance types, if needed. :::note To use the same queue for both head and compute tasks, you must assign sufficient resources to your compute environment. ::: 1. Expand **Additional configuration** and select the `seqera-launchtemplate` from the Launch template drop-down. 1. Configure VPCs, subnets, and security groups on the next page as needed. 1. Review your configuration and select **Create compute environment**. Create this compute environment to use Spot instances for your workflow compute tasks. This compute environment cannot be assigned to the Nextflow head job queue. 1. In the [Batch Console](https://eu-west-1.console.aws.amazon.com/batch/home), select **Create** on the Compute environments page. 1. Select **Amazon EC2** as the compute environment configuration. 1. Enter a name of your choice, and apply the `seqera-servicerole` and `seqera-instancerole`. 1. Select **Enable using Spot instances** to use Spot instances and save computing costs. 1. Select the `seqera-fleetrole` and enter vCPU limits and instance types, if needed. 1. Expand **Additional configuration** and select the `seqera-launchtemplate` from the Launch template drop-down. 1. Configure VPCs, subnets, and security groups on the next page as needed. 1. Review your configuration and select **Create compute environment**. ### Create the Batch queue Create a Batch queue to be associated with each compute environment. :::note You only need to create one queue if you intend to use on-demand instances for your workflow compute tasks. Compute environments with Spot instances require separate queues for the head and compute tasks. ::: 1. Go to the [Batch Console](https://eu-west-1.console.aws.amazon.com/batch/home). 2. Create a new queue. 3. Associate the queue with the head queue compute environment created in the previous section. 4. Save it with a name of your choice. 1. Go to the [Batch Console](https://eu-west-1.console.aws.amazon.com/batch/home). 2. Create a new queue. 3. Associate the queue with the compute queue environment created in the previous section. 4. Save it with a name of your choice. Use the AWS resources created on this page to create your [manual AWS Batch compute environment](../../compute-envs/aws-batch#manual-configuration-of-batch-resources). --- ## Set up advanced Azure Batch compute environments Batch Forge provisions most Azure Batch topologies for you: separate head and worker pools, dedicated head job resources, autoscaling, Microsoft Entra authentication, and private networking. Each part below adds one capability to the setup before it. Work through them in order. The final part covers the cases that still need a pool you create yourself. For every compute environment field, see the [Azure Batch][azure-batch-reference] reference. :::info[**Prerequisites**] You need the following: - An Azure account with permissions to create resources. - The [Azure CLI][install-azure-cli]. - The [Seqera Platform CLI][install-seqera-cli]. ::: ## Set up Azure Batch In the Azure Portal: 1. Create an Azure Storage account with the default settings. 1. In the Storage account, add a single blob container called `work`. This is the [Nextflow working directory][nextflow-working-directory]. 1. Create an Azure Batch account with the default settings and the Batch Managed pool allocation mode. Use the same region as your Storage account, and attach the Storage account to the Batch account when prompted. 1. On the Azure Batch page, select **Quotas**. 1. Select **Request Quota Increase**. 1. For **Quota Type**, select **Batch**, then select **Next**. 1. Select **Enter Details**, then set **Location** to the region of your Batch account. 1. Select **EDv5 Series**. 1. Select **Active jobs and job schedules per Batch account**. 1. Select **Pools per Batch account**. Increase each value to at least the following: - **EDv5 Series**: 192 - **Active jobs and job schedules per Batch account**: 100 - **Pools per Batch account**: 50 ## Set up Seqera Platform In Seqera Platform: - Sign in to your Seqera Platform deployment. - [Create a new organization and workspace][create-org-workspace]. - Add a GitHub credential to the workspace to prevent API rate-limiting issues with GitHub. ## Part 1. Create a Batch Forge compute environment Batch Forge creates and manages the Azure Batch pools for you. By default it creates two pools, one for the Nextflow head job and one for compute tasks, named `tower-pool-{envId}-head` and `tower-pool-{envId}-worker`. The head node does not compete with tasks for resources, and you can size each pool independently. Both pools autoscale based on the number of waiting tasks. Add the Azure Batch account credentials to Seqera Platform: 1. In the Azure Portal, open the Batch account you created and note the Batch account name and region. 1. On the **Keys** tab, find the primary access keys for the Batch account and the Storage account. 1. In your Seqera Platform workspace, select the **Credentials** tab, then select **Add credentials**. 1. Enter a credential name such as `azure-keys`, then select **Azure** for **Provider**. 1. Enter the Batch account name and key, and the Storage account name and key. 1. Select **Create**. Create the compute environment: 1. Select **Compute Environments**, then select **Add compute environment**. 1. Enter a name such as `azure-batch-forge`. 1. For **Provider**, select **Azure Batch**. 1. Select your `azure-keys` credentials. 1. Select the **Location** of your Batch account. 1. Select the `az://work` container in your Storage account. 1. For **Pool mode**, select **Separate head and worker pools**. 1. Set **Config mode** to **Batch Forge**. 1. Under **Head pool**, leave **VM type** and **VMs count** at their defaults, and enable **Autoscale**. 1. Under **Worker pool**, set **VM type** to `Standard_E2ds_v5`, set **VMs count** to `4`, and enable **Autoscale**. 1. (Optional) Set **Head job CPUs** and **Head job memory** to size each Nextflow head job within the head pool. 1. Enable **Dispose resources**. 1. Leave the remaining options at their defaults, then select **Create**. :::tip To reduce pipeline latency, disable **Autoscale** under **Head pool** and set its **VMs count** to `1`. One head node then stays running while the worker pool autoscales independently. An always-on head node costs more but responds faster. The difference is most noticeable on large production pipelines. ::: [Add a pipeline][add-pipeline] named `nextflow-hello` from your workspace Launchpad with the following settings: - Select your Azure Batch compute environment. - For **Pipeline to launch**, enter `https://github.com/nextflow-io/hello`. - For **Work directory**, enter a subdirectory of the `az://work` container in your Storage account. On the Launchpad, select **Launch** next to the pipeline name, complete the launch form, and start the run. ## Part 2. Authenticate with Microsoft Entra Instead of access keys, Seqera Platform can authenticate to Azure Batch and Storage with a Microsoft Entra service principal, and Nextflow can authenticate with a managed identity. You create the service principal and managed identity in Azure, then select them in the compute environment form. Batch Forge configures the pool identity for you. Entra authentication has the following advantages over access keys: - No keys or short-lived access tokens are exchanged. - A service principal holds only the permissions you grant it. - A managed identity scoped to a single resource keeps the Nextflow head job's permissions narrow. - Different compute environments can use different managed identities, each with its own permissions. For more detail, see [Microsoft Entra](https://docs.seqera.io/nextflow/azure#microsoft-entra) in the Nextflow documentation. ### Create a service principal for Seqera Platform In the Azure Portal: 1. [Create an Azure service principal](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal). 1. [Assign roles to the service principal](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). 1. [Create a client secret](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal#option-3-create-a-new-client-secret), then note the client ID, tenant ID, and client secret value. The service principal needs the following role assignments: - **Azure Batch Data Contributor** on the Batch account. - **Storage Blob Data Contributor** on the Storage account. - **Managed Identity Operator** on each managed identity. - **Network Contributor** on the virtual network (VNet), when you use one. For details, see [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference. In Seqera Platform: 1. Select the **Credentials** tab, then select **Add credentials**. 1. Enter the name `entra-keys` and select **Azure** for **Provider**. 1. Select the **Entra** tab, then enter the **Client ID**, **Tenant ID**, and **Client secret**. 1. Select **Create**. ### Create a managed identity for Nextflow In the Azure Portal: 1. [Create a managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?pivots=identity-mi-methods-azp). 1. [Assign the relevant roles to the managed identity](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). For the Nextflow requirements, see [Required role assignments](https://docs.seqera.io/nextflow/azure#required-role-assignments). 1. Note the managed identity client ID and resource ID. In Seqera Platform: 1. Add a new Batch Forge compute environment named `entra-mi`, and select **Azure Batch** for **Provider**. 1. For **Credentials**, select the `entra-keys` service principal credentials. 1. Select the **Location** of your Batch account. 1. In the managed identity fields, enter the client ID and resource ID for both the head and worker pools. You can use the same managed identity for both pools. 1. Configure the remaining fields as in [Part 1](#part-1-create-a-batch-forge-compute-environment). Duplicate the `nextflow-hello` pipeline, save it as `hello-world-entra-mi`, and select the new compute environment. On the Launchpad, select **Launch** next to `hello-world-entra-mi` and start the run. The pipeline runs as before, but authenticates to Azure Batch and Storage with the managed identity. No keys are exchanged. ## Part 3. Attach the pool to a virtual network To connect the Batch pool nodes to a private Azure VNet, enter a **Subnet ID** when you create the Batch Forge compute environment. Batch Forge attaches the subnet to both the head and worker pools, and the worker nodes inherit the same networking. A private VNet restricts network access to the subnet you specify, and data transfer inside the VNet can be faster and cheaper than routing over the public internet. :::note The **Subnet ID** field is only available when you select Entra credentials. See [Entra service principal and managed identity][azure-batch-entra] in the Azure Batch reference. ::: Before you create the compute environment, assign the **Network Contributor** role to the service principal on the VNet. Alternatively, use a custom role that grants `Microsoft.Network/virtualNetworks/subnets/join/action`. 1. Add a new Batch Forge compute environment named `azure-batch-vnet`, and select **Azure Batch** for **Provider**. 1. For **Credentials**, select the `entra-keys` service principal credentials. 1. Select the **Location** of your Batch account. 1. For **Subnet ID**, enter the full Azure Resource Manager (ARM) subnet resource ID: ``` /subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.Network/virtualNetworks/{vnetName}/subnets/{subnetName} ``` 1. Configure the remaining fields as in [Part 1](#part-1-create-a-batch-forge-compute-environment). Duplicate the `nextflow-hello` pipeline, save it as `hello-world-vnet`, and select the new compute environment. Select **Launch** to run the pipeline on the pool attached to the VNet. ## Part 4. Use a pre-existing Azure Batch pool Some setups still need a pool you create yourself: low-priority (spot) worker nodes, custom VM images, or Azure Batch features that Batch Forge does not expose. Batch Forge provisions dedicated (on-demand) nodes only. In these cases, create the pool in Azure and select it in a **Manual** compute environment. Manual pools can use the Batch Managed or User Subscription pool allocation mode. Batch Forge supports Batch Managed only. Manual mode does not attach a subnet to a pre-existing pool. Configure the VNet and subnet on each pool as you create it. For the full pool configuration (identity, OS image, autoscale formula, start task, and networking), see [Create a Nextflow-compatible Azure Batch pool][azure-batch-manual-pool] in the Azure Batch reference. Then create a Manual compute environment that points at the pool name. To run compute tasks on low-priority nodes, create one dedicated pool and one low-priority pool, then route compute tasks to the low-priority pool with the `process.queue` directive. See [Use separate Batch pools for head and compute nodes][azure-separate-pools]. :::note The Nextflow autopool feature (`azure.batch.autoPoolMode` and `azure.batch.allowPoolCreation`) is deprecated and unused by Seqera Platform. Use the Batch Forge head and worker pools instead. ::: ## Clear up resources After the setup and the runs are complete, delete the pipelines and compute environments from Seqera Platform. In Azure, deleting the Batch account deletes all its pools, jobs, and tasks. You can then delete the Storage account. To keep the Azure resources, remove each pool in the Batch account and mark any active jobs as terminated to free up quota on the Batch account. [install-azure-cli]: https://learn.microsoft.com/en-us/cli/azure/install-azure-cli [install-seqera-cli]: /platform-cli/installation [nextflow-working-directory]: https://docs.seqera.io/nextflow/cache-and-resume#work-directory [create-org-workspace]: ../../getting-started/workspace-setup [add-pipeline]: ../../getting-started/quickstart-demo/add-pipelines#add-from-the-launchpad [azure-batch-reference]: ../../compute-envs/azure-batch [azure-batch-entra]: ../../compute-envs/azure-batch#entra-service-principal-and-managed-identity [azure-batch-manual-pool]: ../../compute-envs/azure-batch#create-a-nextflow-compatible-azure-batch-pool [azure-separate-pools]: ../../troubleshooting_and_faqs/azure_troubleshooting#use-separate-batch-pools-for-head-and-compute-nodes --- ## Seqera Platform Monitoring Seqera Platform has built-in observability metrics. Enable observability metrics by adding `prometheus` to the `MICRONAUT_ENVIRONMENTS` environment variable. This exposes a Prometheus endpoint at `/prometheus` on the default listen port (e.g., `http://localhost:8080/prometheus`). Combined with infrastructure monitoring tools such as Node Exporter, you can monitor relevant metrics across your deployment. ## Alerting recommendations ### Critical alerts - `jvm_memory_used_bytes{area="heap"}` > 90% of `jvm_memory_max_bytes` - `process_files_open_files` > 90% of `process_files_max_files` - `logback_events_total{level="error"}` rate > threshold - `tower_logs_errors_1minCount` > 0 - HTTP 5xx errors > 5% of total requests - `jdbc_connections_active` > 90% of `jdbc_connections_max` - Any pods in Failed/Unknown state for > 5 minutes ### Warning alerts - `jvm_gc_pause_seconds_max` > 1 second - `jvm_gc_live_data_size_bytes` approaching `jvm_gc_max_data_size_bytes` - Heap usage > 85% of max heap - `executor_queued_tasks` > threshold - Executor utilization > 90% - `hibernate_optimistic_failures_total` rate increasing - `hibernate_query_executions_max_seconds` > 5 seconds - `http_server_requests_seconds` p99 > acceptable latency - Redis cache hit rate < 70% - Hibernate query cache hit rate < 60% - Growing gap between `credits_estimation_workflow_added_total` and `credits_estimation_workflow_ended_total` - `hibernate_sessions_open_total` >> `hibernate_sessions_closed_total` over time ## Quick reference: Metrics by troubleshooting scenario | Issue | Key Metrics to Check | | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | | **Slow application response** | `http_server_requests_seconds` (latency), `jvm_gc_pause_seconds_max`, `hibernate_query_executions_max_seconds`, `executor_active_threads` | | **Out of memory errors** | `jvm_memory_used_bytes`, `jvm_gc_pause_seconds`, `jvm_gc_live_data_size_bytes`, `jvm_buffer_memory_used_bytes` | | **Database performance** | `hibernate_query_executions_max_seconds`, `jdbc_connections_active`, `hibernate_transactions_total`, cache hit rates | | **High CPU usage** | `process_cpu_usage`, `system_cpu_usage`, `jvm_threads_live_threads`, `executor_active_threads` | | **Connection exhaustion** | `jdbc_connections_active`, `jdbc_connections_max`, `hibernate_sessions_open_total` vs `hibernate_sessions_closed_total` | | **Cache issues** | Redis hit rate, `hibernate_cache_query_requests_total`, `cache_gets_total`, `cache_evictions_total` | | **Workflow processing delays** | `credits_estimation_workflow_*`, `credits_estimation_task_*`, `executor_queued_tasks`, `tower_logs_errors_*` | | **Thread starvation** | `executor_active_threads`, `executor_queued_tasks`, `jvm_threads_states_threads{state="blocked"}` | | **Memory leaks** | `jvm_memory_used_bytes` trending up, `jvm_gc_live_data_size_bytes` growing, `jvm_classes_loaded_classes` growing | | **GC pressure** | `jvm_gc_pause_seconds_max`, `jvm_gc_memory_promoted_bytes_total`, time in GC vs application time | ## Key metrics to monitor ### Seqera Platform-specific metrics #### Studios metrics | Metric | Description | | ------------------------------------------------ | -------------------------------------| | `data_studio_startup_time_failure_seconds_sum` | Time for failed Studio startups | | `data_studio_startup_time_failure_seconds_count` | Failed Studio startup count | Track Studio startup performance to identify environment provisioning issues. Slow or failing startups impact user productivity. **Average startup time by tool** ```shell sum by (tool) (increase(data_studio_startup_time_success_seconds_sum{app="backend", namespace="$namespace"}[$__rate_interval])) / sum by (tool) (increase(data_studio_startup_time_success_seconds_count{app="backend", namespace="$namespace"}[$__rate_interval])) ``` **Failed startup rate** ```shell rate(data_studio_startup_time_failure_seconds_count{namespace="$namespace"}[$__rate_interval]) ``` #### Error tracking | Metric | Description | | ------------------------------ | ------------------------- | | `tower_logs_errors_10secCount` | Errors in last 10 seconds | | `tower_logs_errors_1minCount` | Errors in last minute | | `tower_logs_errors_5minCount` | Errors in last 5 minutes | Monitor application errors across different time windows. Rolling error counts help identify transient issues versus sustained problems. **Recent error counts** ```shell tower_logs_errors_10secCount{namespace="$namespace"} tower_logs_errors_1minCount{namespace="$namespace"} tower_logs_errors_5minCount{namespace="$namespace"} ``` **Log events by severity level** ```shell rate(logback_events_total{namespace="$namespace"}[$__rate_interval]) ``` ### Infrastructure resources #### CPU usage Monitor container CPU consumption against requested resources to identify capacity issues or inefficient resource allocation. **Backend CPU usage** ```shell rate(container_cpu_usage_seconds_total{container="backend", namespace="$namespace"}[$__rate_interval]) ``` **Compare against requested resources** to determine if the container is over or under-provisioned: ```shell max(kube_pod_container_resource_requests{container="backend", namespace="$namespace", resource="cpu"}) ``` #### Memory usage Track working set memory, committed memory, and limits to prevent OOM conditions. **Backend memory working set** shows actual memory in use: ```shell container_memory_working_set_bytes{container="backend", namespace="$namespace"} ``` **Memory requests and limits** define the bounds for container memory allocation: ```shell max(kube_pod_container_resource_requests{container="backend", namespace="$namespace", resource="memory"}) max(kube_pod_container_resource_limits{container="backend", namespace="$namespace", resource="memory"}) ``` ### HTTP server requests | Metric | Description | | ------------------------------------------ | ------------------------------------------------- | | `http_server_requests_seconds_count` | Total request count by method, status, and URI | | `http_server_requests_seconds_sum` | Total request duration by method, status, and URI | | `http_server_requests_seconds_max` | Maximum request duration | | `http_server_requests_seconds` (quantiles) | Request latency percentiles (p50, p95, p99, p999) | HTTP metrics reveal application throughput, error rates, and latency patterns. These are essential for understanding user-facing performance. **Total request throughput** shows overall API activity: ```shell sum(rate(http_server_requests_seconds_count{app="backend", namespace="$namespace"}[$__rate_interval])) ``` **Error rate (4xx and 5xx responses)** indicates client errors and server failures: ```shell sum(rate(http_server_requests_seconds_count{app="backend", namespace="$namespace", status=~"[45].."}[$__rate_interval])) ``` **Average latency per endpoint** helps identify slow API paths: ```shell sum by (method, uri) (rate(http_server_requests_seconds_sum{app="backend", namespace="$namespace"}[$__rate_interval])) / sum by (method, uri) (rate(http_server_requests_seconds_count{app="backend", namespace="$namespace"}[$__rate_interval])) ``` **Top 10 endpoints by time spent** highlights where server time is consumed for optimization efforts: ```shell topk(10, sum by(method, uri) (rate(http_server_requests_seconds_sum{namespace="$namespace", app="backend"}[$__rate_interval]))) ``` ### HTTP client requests | Metric | Description | | ------------------------------------ | --------------------------------- | | `http_client_requests_seconds_count` | Outbound request count | | `http_client_requests_seconds_sum` | Total outbound request duration | | `http_client_requests_seconds_max` | Maximum outbound request duration | Monitor external API calls and integrations. Slow or failing outbound requests can cascade into application performance issues. **Outbound request rate** ```shell rate(http_client_requests_seconds_count{namespace="$namespace"}[$__rate_interval]) ``` **Average outbound request duration** ```shell rate(http_client_requests_seconds_sum{namespace="$namespace"}[$__rate_interval]) / rate(http_client_requests_seconds_count{namespace="$namespace"}[$__rate_interval]) ``` **Maximum outbound request duration** identifies slow external dependencies: ```shell http_client_requests_seconds_max{namespace="$namespace"} ``` ### JVM memory metrics | Metric | Description | | ------------------------------ | -------------------------------------------------------- | | `jvm_buffer_memory_used_bytes` | Memory used by JVM buffer pools (direct, mapped) | | `jvm_memory_used_bytes` | Amount of used memory by area (heap/non-heap) and region | | `jvm_memory_committed_bytes` | Memory committed for JVM use | | `jvm_memory_max_bytes` | Maximum memory available for memory management | | `jvm_gc_live_data_size_bytes` | Size of long-lived heap memory pool after reclamation | | `jvm_gc_max_data_size_bytes` | Max size of long-lived heap memory pool | JVM memory metrics are critical for preventing OutOfMemoryErrors and identifying memory leaks. Monitor both heap (Java objects) and non-heap (metaspace, code cache) regions. **Heap memory usage** shows memory used for Java objects: ```shell jvm_memory_used_bytes{app="backend", namespace="$namespace", area="heap"} jvm_memory_committed_bytes{app="backend", namespace="$namespace", area="heap"} jvm_memory_max_bytes{app="backend", namespace="$namespace", area="heap"} ``` **Non-heap memory** includes metaspace and code cache: ```shell jvm_memory_used_bytes{app="backend", namespace="$namespace", area="nonheap"} jvm_memory_committed_bytes{app="backend", namespace="$namespace", area="nonheap"} jvm_memory_max_bytes{app="backend", namespace="$namespace", area="nonheap"} ``` **Heap usage percentage** provides a quick health indicator. Alert when this exceeds 85%: ```shell sum(jvm_memory_used_bytes{area="heap"}) / sum(jvm_memory_max_bytes{area="heap"}) * 100 ``` **Direct buffer usage** is important for Netty-based applications. High usage can cause native memory issues: ```shell jvm_buffer_memory_used_bytes{namespace="$namespace", app="backend", id="direct"} jvm_buffer_total_capacity_bytes{namespace="$namespace", app="backend", id="direct"} ``` ### JVM garbage collection | Metric | Description | | ------------------------------------- | ----------------------------------------- | | `jvm_gc_pause_seconds_sum` | Total time spent in GC pauses | | `jvm_gc_pause_seconds_count` | Number of GC pause events | | `jvm_gc_pause_seconds_max` | Maximum GC pause duration | | `jvm_gc_memory_allocated_bytes_total` | Total bytes allocated in young generation | | `jvm_gc_memory_promoted_bytes_total` | Bytes promoted to old generation | Garbage collection metrics reveal memory pressure and its impact on application responsiveness. Long GC pauses cause request latency spikes. **Average GC pause duration** should remain low (under 100ms for most applications): ```shell rate(jvm_gc_pause_seconds_sum{app="backend", namespace="$namespace"}[$__rate_interval]) / rate(jvm_gc_pause_seconds_count{app="backend", namespace="$namespace"}[$__rate_interval]) ``` **Maximum GC pause** identifies worst-case latency impact. Alert if this exceeds 1 second: ```shell jvm_gc_pause_seconds_max{app="backend", namespace="$namespace"} ``` **Live data size after GC** shows long-lived objects. If this grows over time, you may have a memory leak: ```shell jvm_gc_live_data_size_bytes{app="backend", namespace="$namespace"} ``` **Memory allocation and promotion rates** indicate object creation patterns. High promotion rates suggest objects are living longer than expected: ```shell rate(jvm_gc_memory_allocated_bytes_total{app="backend", namespace="$namespace"}[$__rate_interval]) rate(jvm_gc_memory_promoted_bytes_total{app="backend", namespace="$namespace"}[$__rate_interval]) ``` ### JVM threads | Metric | Description | | ---------------------------- | ----------------------------------------------------------------- | | `jvm_threads_live_threads` | Current number of live threads (daemon + non-daemon) | | `jvm_threads_daemon_threads` | Current number of daemon threads | | `jvm_threads_peak_threads` | Peak thread count since JVM start | | `jvm_threads_states_threads` | Thread count by state (runnable, blocked, waiting, timed-waiting) | Thread metrics help identify deadlocks, thread pool exhaustion, and concurrency issues. **Thread counts** show overall thread activity: ```shell jvm_threads_live_threads{app="backend", namespace="$namespace"} jvm_threads_daemon_threads{app="backend", namespace="$namespace"} jvm_threads_peak_threads{app="backend", namespace="$namespace"} ``` **Thread states** reveal blocking issues. High blocked thread counts indicate lock contention: ```shell jvm_threads_states_threads{app="backend", namespace="$namespace"} ``` ### JVM classes | Metric | Description | | ------------------------------------ | -------------------------------------- | | `jvm_classes_loaded_classes` | Currently loaded classes | | `jvm_classes_unloaded_classes_total` | Total classes unloaded since JVM start | Class loading metrics help identify class loader leaks or excessive dynamic class generation. **Loaded classes** should stabilize after startup. Continuous growth may indicate a class loader leak: ```shell jvm_classes_loaded_classes{namespace="$namespace", app="backend"} ``` **Class unload rate** ```shell rate(jvm_classes_unloaded_classes_total{namespace="$namespace", app="backend"}[$__rate_interval]) ``` ### Process metrics | Metric | Description | | ---------------------------- | ------------------------------------ | | `process_cpu_usage` | Recent CPU usage for the JVM process | | `process_cpu_time_ns_total` | Total CPU time used by the JVM | | `process_files_open_files` | Open file descriptor count | | `process_files_max_files` | Maximum file descriptor limit | | `process_uptime_seconds` | JVM uptime | | `process_start_time_seconds` | Process start time (unix epoch) | Process-level metrics provide visibility into resource consumption and system limits. **JVM process CPU usage** ```shell process_cpu_usage{namespace="$namespace"} ``` **Open file descriptors** should be monitored against limits. Exhaustion causes connection failures: ```shell process_files_open_files{namespace="$namespace"} ``` **File descriptor utilization percentage** - alert when this exceeds 90%: ```shell (process_files_open_files{namespace="$namespace"} / process_files_max_files{namespace="$namespace"}) * 100 ``` **Process uptime** helps identify restart events. Low uptime may indicate stability issues: ```shell process_uptime_seconds{namespace="$namespace"} ``` ### System metrics | Metric | Description | | ------------------------ | ------------------------------------- | | `system_cpu_usage` | System-wide CPU usage | | `system_cpu_count` | Number of processors available to JVM | | `system_load_average_1m` | 1-minute load average | System metrics provide host-level context for application performance. **System-wide CPU usage** ```shell system_cpu_usage{namespace="$namespace"} ``` **System load average** should remain below the CPU count for healthy systems: ```shell system_load_average_1m{namespace="$namespace"} ``` **Available CPU count** ```shell system_cpu_count{namespace="$namespace"} ``` ### Executor thread pools | Metric | Description | | -------------------------------- | ---------------------------------------------------------- | | `executor_active_threads` | Currently active threads by pool (io, blocking, scheduled) | | `executor_pool_size_threads` | Current thread pool size | | `executor_pool_max_threads` | Maximum allowed threads in pool | | `executor_queued_tasks` | Tasks queued for execution | | `executor_completed_tasks_total` | Total completed tasks | | `executor_seconds_sum` | Total execution time | Thread pool metrics reveal concurrency bottlenecks. Saturated pools cause request queuing and increased latency. **Thread pool utilization percentage** - high utilization indicates the pool is near capacity: ```shell executor_active_threads{service="backend", namespace="$namespace", name!="scheduled"} / executor_pool_size_threads{service="backend", namespace="$namespace", name!="scheduled"} ``` **Cron scheduled executor utilization** ```shell executor_active_threads{service="cron", namespace="$namespace", name="scheduled"} / executor_pool_size_threads{service="cron", namespace="$namespace", name="scheduled"} ``` **Queued tasks** indicate backlog. Growing queues suggest the pool cannot keep up with demand: ```shell executor_queued_tasks{app="backend", namespace="$namespace"} ``` **Task completion rate** ```shell rate(executor_completed_tasks_total{namespace="$namespace"}[$__rate_interval]) ``` ### Cache metrics | Metric | Description | | ----------------------- | ----------------------------------- | | `cache_size` | Number of entries in cache | | `cache_gets_total` | Cache hits and misses by cache name | | `cache_puts_total` | Cache entries added | | `cache_evictions_total` | Cache eviction count | Cache effectiveness directly impacts database load and response times. Low hit rates indicate caching issues. **Redis cache hit rate** - should be above 70% for effective caching: ```shell avg(irate(redis_keyspace_hits_total{app="platform-redis-exporter"}[$__rate_interval]) / (irate(redis_keyspace_misses_total{app="platform-redis-exporter"}[$__rate_interval]) + irate(redis_keyspace_hits_total{app="platform-redis-exporter"}[$__rate_interval]))) ``` **Cache size by name** ```shell cache_size{namespace="$namespace"} ``` **Cache operation rates** ```shell rate(cache_gets_total{namespace="$namespace"}[$__rate_interval]) rate(cache_puts_total{namespace="$namespace"}[$__rate_interval]) rate(cache_evictions_total{namespace="$namespace"}[$__rate_interval]) ``` ### Hibernate/Database metrics | Metric | Description | | ---------------------------------------- | ---------------------------------------------------- | | `hibernate_sessions_open_total` | Total sessions opened | | `hibernate_sessions_closed_total` | Total sessions closed | | `hibernate_connections_obtained_total` | Database connections obtained | | `hibernate_query_executions_total` | Total queries executed | | `hibernate_query_executions_max_seconds` | Slowest query time | | `hibernate_entities_inserts_total` | Entity insert operations | | `hibernate_entities_updates_total` | Entity update operations | | `hibernate_entities_deletes_total` | Entity delete operations | | `hibernate_entities_loads_total` | Entity load operations | | `hibernate_transactions_total` | Transaction count | | `hibernate_flushes_total` | Session flush count | | `hibernate_optimistic_failures_total` | Optimistic lock failures (StaleObjectStateException) | Database metrics reveal query performance, connection management, and transaction health. **Session operations** - open and closed counts should be roughly equal. A growing gap indicates session leaks: ```shell rate(hibernate_sessions_open_total{app="backend", namespace="$namespace"}[$__rate_interval]) rate(hibernate_sessions_closed_total{app="backend", namespace="$namespace"}[$__rate_interval]) ``` **Connection acquisition rate** ```shell rate(hibernate_connections_obtained_total{app="backend", namespace="$namespace"}[$__rate_interval]) ``` **Query execution rate** ```shell rate(hibernate_query_executions_total{app="backend", namespace="$namespace"}[$__rate_interval]) ``` **Query latency by type** helps identify slow queries for optimization: ```shell sum by (query) (rate(hibernate_query_execution_total_seconds_sum{app="backend", namespace="$namespace"}[$__rate_interval])) / sum by (query) (rate(hibernate_query_execution_total_seconds_count{app="backend", namespace="$namespace"}[$__rate_interval])) ``` **Slowest query time** - alert if this exceeds 5 seconds: ```shell hibernate_query_executions_max_seconds{app="backend", namespace="$namespace"} ``` **Entity operation rates** show database write patterns: ```shell rate(hibernate_entities_inserts_total{app="backend", namespace="$namespace"}[$__rate_interval]) rate(hibernate_entities_updates_total{app="backend", namespace="$namespace"}[$__rate_interval]) rate(hibernate_entities_deletes_total{app="backend", namespace="$namespace"}[$__rate_interval]) rate(hibernate_entities_loads_total{app="backend", namespace="$namespace"}[$__rate_interval]) ``` **Transaction success/failure rate** ```shell sum by (result) (rate(hibernate_transactions_total{app="backend", namespace="$namespace"}[$__rate_interval])) ``` **Optimistic lock failures** indicate concurrent modification conflicts. High rates suggest contention issues: ```shell rate(hibernate_optimistic_failures_total{app="backend", namespace="$namespace"}[$__rate_interval]) ``` ### Connection pool metrics | Metric | Description | | ------------------------- | ---------------------------- | | `jdbc_connections_active` | Active database connections | | `jdbc_connections_max` | Maximum connection pool size | | `jdbc_connections_min` | Minimum connection pool size | | `jdbc_connections_usage` | Connection pool usage | Connection pool metrics prevent connection exhaustion during traffic bursts. **Active connections vs pool limits** - alert when active connections approach the maximum: ```shell sum(jdbc_connections_active{app="backend", namespace="$namespace"}) sum(jdbc_connections_max{app="backend", namespace="$namespace"}) sum(jdbc_connections_min{app="backend", namespace="$namespace"}) sum(jdbc_connections_usage{app="backend", namespace="$namespace"}) ``` ### Hibernate cache metrics Hibernate caching reduces database load. Monitor hit rates to ensure caches are effective. **Query cache hit rate** - should exceed 60%: ```shell sum(increase(hibernate_cache_query_requests_total{app="backend", namespace="$namespace", result="hit"}[$__rate_interval])) / sum(increase(hibernate_cache_query_requests_total{app="backend", namespace="$namespace"}[$__rate_interval])) ``` **Query plan cache hit rate** ```shell sum(increase(hibernate_cache_query_plan_total{app="backend", namespace="$namespace", result="hit"}[$__rate_interval])) / sum(increase(hibernate_cache_query_plan_total{app="backend", namespace="$namespace"}[$__rate_interval])) ``` **Second level cache hit rate by region** ```shell sum by (region) (increase(hibernate_second_level_cache_requests_total{app="backend", namespace="$namespace", result="hit"}[$__rate_interval])) / sum by (region) (increase(hibernate_second_level_cache_requests_total{app="backend", namespace="$namespace"}[$__rate_interval])) ``` ### Logging metrics | Metric | Description | | ---------------------- | ----------------------------------------------------- | | `logback_events_total` | Log events by level (debug, info, warn, error, trace) | Log event metrics provide early warning of application issues. **Error rate** - track error log frequency for anomaly detection: ```shell rate(logback_events_total{level="error"}[5m]) ``` ### Kubernetes health Monitor pod health to catch deployment or infrastructure issues early. **Pods in unhealthy states** ```shell sum by (namespace, pod) (kube_pod_status_phase{phase=~"Pending|Unknown|Failed", namespace!="wave-build"}) > 0 ``` --- ## Legacy Seqera container image registries :::caution The `cr.seqera.io` container registry is the default Seqera Enterprise container image registry from version 22.4. Using the AWS ECR Seqera container registry in existing installations is still supported but will be deprecated on June 1, 2025. ::: Seqera publishes legacy Seqera Enterprise containers to a private Elastic Container Registry (ECR) on AWS. Retrieve them with the following steps: 1. **Provide Seqera with your AWS Account ID.** Supply this value to the Seqera representative managing your onboarding and wait for confirmation that it has been added to the ECR repository policy as an approved Principal. 2. **Retrieve a local copy of the container.** With the `docker compose` deployment method, you must retrieve container copies for local use: 1. Install [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) on the target machine. 2. Configure the AWS CLI with an IAM User with at least these privileges: ```bash ecr:BatchGetImage ecr:GetAuthorizationToken ecr:GetDownloadUrlForLayer ``` 3. Authenticate Docker against the Seqera ECR: ```bash # AWS CLI v2 aws ecr get-login-password --region eu-west-1 | \ docker login --username AWS --password-stdin 195996028523.dkr.ecr.eu-west-1.amazonaws.com # AWS CLI v1 $(aws ecr get-login --registry-ids 195996028523 --region eu-west-1 --no-include-email) ``` 4. Pull the containers to your machine: ```bash export REPOSITORY_URL="195996028523.dkr.ecr.eu-west-1.amazonaws.com/nf-tower-enterprise" export TAG="v22.3.1" docker pull ${REPOSITORY_URL}/backend:${TAG} docker pull ${REPOSITORY_URL}/frontend:${TAG} ``` --- ## Email Configure email-based passwordless authentication for Seqera Platform. This is the default authentication method that allows users to sign in using their email address. Email authentication provides a passwordless login experience where users: 1. Enter their email address on the login page 2. Receive an email containing a temporary access link 3. Select the link to authenticate and access Platform New users are automatically registered on their first login if their email address matches the trusted email patterns. The access link contains a time-limited token that expires after use. :::info Prerequisites Before enabling email authentication, you need: - A configured SMTP server for sending authentication emails - Valid SMTP credentials with permission to send emails ::: ## Configure SMTP Email authentication requires an SMTP server to send login links. Add the following environment variables to your Seqera configuration: | Variable | Description | Required | | :-------------------- | :------------------------------------------------------- | :---------------------------- | | `TOWER_SMTP_HOST` | SMTP server hostname | Yes | | `TOWER_SMTP_PORT` | SMTP server port (e.g., 587 for TLS, 25 for unencrypted) | Yes | | `TOWER_SMTP_USER` | SMTP username | If authentication is required | | `TOWER_SMTP_PASSWORD` | SMTP password | If authentication is required | | `TOWER_SMTP_AUTH` | Set to `true` to enable SMTP authentication | No (default: `false`) | | `TOWER_CONTACT_EMAIL` | Sender email address for authentication emails | Yes | :::tip For development, you can use a local SMTP server like [Mailpit](https://github.com/axllent/mailpit) to test email authentication without sending real emails. ::: ## Restrict access By default, all email addresses are allowed to authenticate. To restrict access to specific email addresses or domains, configure a trusted email list in `tower.yml`: ```yaml tower: trustedEmails: - "*@your-company.com" - "*@partner-company.com" - "external-user@example.com" ``` Pattern matching: - `*@domain.com` - allows all emails from the domain - `*@*.example.com` - allows all subdomains - `user@domain.com` - allows a specific email address - `user+*@domain.com` - allows plus addressing (e.g., `user+tag@domain.com`) When `trustedEmails` is not specified, all email addresses are trusted and can create accounts. See [User access allow list](./overview#user-access-allow-list) for more information. ## Disable email authentication To disable email authentication when other authentication providers (OAuth, OIDC, etc.) are configured, add the following environment variable: | Variable | Description | | :------------------------- | :----------------------------------------- | | `TOWER_AUTH_DISABLE_EMAIL` | Set to `true` to disable email-based login | :::warning Email authentication can only be disabled when at least one other authentication provider is configured. Users will not be able to log in if email authentication is disabled without an alternative authentication method. ::: --- ## GitHub Configure GitHub as a single sign-on (SSO) provider for Seqera Platform. :::info Prerequisites Before you begin, you need: - A GitHub organization - Permission to create OAuth Apps in your organization Ensure you know how to create a GitHub OAuth app. See GitHub's documentation on [creating an OAuth app](https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/creating-an-oauth-app) for more information. ::: ## Create a GitHub OAuth App 1. In **Profile > Settings > Developer settings**, select **OAuth Apps**. 2. Select **New OAuth App**. 3. Complete the required fields. In the **Authorization callback URL** field, enter `https:///oauth/callback/github` (must be HTTPS) - replace `` with your enterprise installation hostname. 4. Note your **Client ID**. 5. Generate a client secret, then note your **Client secret**. ## Configure Seqera Add the following environment variables to your Seqera configuration: | Variable | Description | | :------- | :---------- | | `TOWER_GITHUB_CLIENT` | The client ID from step 4 | | `TOWER_GITHUB_SECRET` | The client secret from step 5 | ## Restrict access To restrict access to specific email addresses or domains, configure an allow list in `tower.yml`: ```yaml tower: auth: github: allow-list: - "*@your-company.example.com" - "specific-user@another-company.example.net" ``` See [User access allow list](./overview#user-access-allow-list) for more information. --- ## Google Configure Google as a single sign-on (SSO) provider for Seqera Platform. :::info Prerequisites Before you begin, you need: - A Google Cloud account - Permission to create OAuth credentials in the Google Cloud console Ensure you know how to create Google OAuth credentials. See Google's documentation on [setting up OAuth 2.0](https://support.google.com/cloud/answer/6158849) for more information. ::: ## Create Google OAuth credentials 1. In the [Google Cloud console](https://console.developers.google.com), create a new project or select an existing one. 2. Go to **APIs & Services > Credentials**. 3. Select **Create credentials > OAuth client ID**. 4. Select **Web Application** as the application type. 5. Add your redirect URI: `https:///oauth/callback/google` (must be HTTPS) - replace `` with your enterprise installation hostname. 6. Note your **Client ID** and **Client secret**. ## Configure Seqera Add the following environment variables to your Seqera configuration: | Variable | Description | | :------- | :---------- | | `TOWER_GOOGLE_CLIENT` | The client ID from step 6 | | `TOWER_GOOGLE_SECRET` | The client secret from step 6 | ## Restrict access To restrict access to specific email addresses or domains, configure an allow list in `tower.yml`: ```yaml tower: auth: google: allow-list: - "*@your-company.example.com" - "specific-user@another-company.example.net" ``` See [User access allow list](./overview#user-access-allow-list) for more information. --- ## IdP claim mapping For IdP-delegated teams to evaluate correctly at login, the tokens your identity provider sends to Platform must include a `groups` claim. This page lists the per-IdP configuration steps for the supported providers. Enterprise reads the IdP's tokens directly. ## OIDC providers ### Okta 1. In the Okta administrator console, open **Security**, then **API**, then **Authorization Servers**. 2. Select the authorization server backing your Seqera application (typically `default`). 3. Open **Claims**, then **Add claim**. 4. Set: - **Name**: `groups` - **Include in token type**: **ID Token** (and **Access Token** if you use access tokens for downstream services) - **Value type**: **Groups** - **Filter**: Match the groups you want exposed (`Matches regex .*` to expose all of them). 5. Select **Save**. :::note Okta only includes the `groups` claim in the token when the client requests a matching scope. Add `groups` to the scopes Seqera requests by setting `TOWER_OIDC_SCOPES=openid,email,profile,groups`. See [OpenID Connect](../oidc#configure-seqera) for details. ::: ### Entra ID Entra ID requires an app-registration change and attention to the format Entra emits. 1. In the Azure portal, open the app registration that backs your Platform connection. 2. Open **Token configuration**, then **Add groups claim**. 3. Select the group types you want emitted (typically **Security groups**). 4. Under **Customize token properties by type**, choose whether to emit **Group ID** (object GUIDs) or **sAMAccountName** (display names where supported). 5. Confirm via Entra ID's **Token Preview** that a sample sign-in includes the `groups` claim. :::caution With **Group ID** selected, Entra ID emits group object GUIDs. You have two options: - Use the GUID values directly as the catalog identifier and the **IdP Group** field on each team. This works but makes the catalog harder to read. - Configure Entra ID to emit display names instead. Set **sAMAccountName** as the source where supported, or post-process via a custom claims policy. The GUID and the display name don't both flow at the same time, so pick one approach for your tenant and stick with it. ::: ## Verify the mapping After saving the IdP changes, confirm the claim is reaching Platform:: 1. Sign in to Platform as a test user via SSO. 2. In your Platform instance logs, look for the SSO callback log line. It records the full claim set received. 3. Confirm the `groups` claim is present and contains the expected group identifiers. :::caution An absent or empty `groups` claim is treated as "no groups": the user is removed from every team they joined through delegation. Only a malformed `groups` claim (a value that is not a list) is ignored and leaves existing memberships unchanged. ::: --- ## Manage your IdP group catalog Platform maintains a per-organization catalog of identity provider (IdP) groups. Groups appear in the catalog as soon as they're synced from the IdP or added manually. They don't depend on any user having signed in. Use the table below to choose the path that fits your IdP. | IdP | Recommended path | Setup guide | |-----|------------------|-------------| | Okta | SCIM push | [SCIM provisioning with Okta](./scim-okta) | | Entra ID | SCIM push | [SCIM provisioning with Entra ID](./scim-entra-id) | :::info[Other identity providers] SCIM-based provisioning is supported for Okta and Microsoft Entra ID. With these providers, group membership syncs automatically, including lifecycle events (joiners, movers, leavers). Other OIDC or SAML identity providers can authenticate users through Auth0, but group membership doesn't sync automatically. An admin must update memberships in Seqera as users join, move, or leave. If you use Google Workspace, Keycloak, Ping, OneLogin, or another OIDC/SAML provider and want to delegate team membership, contact your Seqera account team to discuss your setup. ::: ## SCIM push If your IdP supports SCIM 2.0 group provisioning, Platform exposes a per-organization SCIM endpoint that the IdP can push to. Group create, rename, and delete events flow through automatically, and the catalog stays in sync without administrator intervention. To set up SCIM: 1. In Platform, open **Organization settings > Group mapping**. 2. Copy the **SCIM endpoint URL** and the generated **bearer token**. 3. Configure these values in your IdP's SCIM provisioning settings. 4. Trigger an initial sync from the IdP or wait until the IdP performs an scheduled sync. After the sync completes, the catalog displays every group your IdP shared, and the **Linked team** drop-down on **Group mapping > IdP groups** is populated. :::caution Treat the SCIM bearer token like a password. It grants write access to your organization's group catalog. If the token is compromised, rotate it immediately using **Rotate** in the **Group mapping** panel. The previous token is revoked atomically. ::: ## Manual entry To add a group manually: 1. In Platform, open **Organization settings > Group mapping**. 2. Select **Add group manually**. 3. Enter the group identifier exactly as it appears in your IdP's `groups` claim. 4. Select **Save**. To delete a manually-entered group, select **Delete** on its row. If any delegated team references the group, its members are immediately purged. :::info A manually-entered group is automatically promoted to SCIM-managed if your IdP later pushes the same group via SCIM. The promotion happens in place. The catalog row is reused, and any delegated teams that reference it continue to work without interruption. After promotion, the row's lifecycle is fully driven by SCIM, and the manual **Delete** action is no longer available. The row is removed when your IdP issues a SCIM `DELETE`. ::: ## Remove catalog entry When a group is removed from the catalog, by SCIM `DELETE`, manual deletion, or IdP-side rename detection, the following happens asynchronously: - The catalog row is removed. - Every delegated team that referenced the group has its delegation-driven members purged. The team's other settings (name, workspace assignments, role) are preserved. - If a group is deleted on the IdP side, the team's membership can be reset by setting its **IdP Group** field to a different group, or clearing the field to convert the team back to manual management. ## Multi-organization deployments On Enterprise instances that host more than one organization, group display names must be unique across all organizations on the instance. If you add a group that conflicts with another organization's catalog entry, it will fail with a `409 Conflict`. See [Multi-organization routing](../multi-org-routing) for more information. --- ## SCIM provisioning with Entra ID Configure Microsoft Entra ID (formerly Azure AD) to push your tenant's groups to Platform over SCIM 2.0. Once provisioning is enabled, the groups you assign to your Seqera Enterprise application appear in Platform's IdP group catalog and stay in sync with renames, additions, and deletions automatically. :::info[**Prerequisites**]{#prerequisites} You will need the following to get started: - An Entra ID application registration configured as your Platform SSO connection. See [Entra ID authentication](../../oidc). SCIM provisioning requires a separate enterprise application that you create in this guide. - Organization owner access to your Platform organization. - Administrator access to your Entra ID tenant with permission to manage application provisioning. ::: ## Get the Platform SCIM connection details 1. In Platform, open **Organization settings > Group mapping**. 2. Copy the **SCIM endpoint URL**. It has the form `https:///orgs//scim/v2`. 3. Select **Generate token** to issue a SCIM bearer token. Copy your bearer token immediately. You can't view it again after closing the dialog. :::caution The bearer token grants write access to your group catalog. Store it in a secrets manager and rotate it on a schedule. To rotate, generate a new token in Seqera and update Entra ID's configuration. The previous token is revoked when the new token is issued. ::: ## Create a provisioning application :::caution Entra ID does not support automatic provisioning on applications created through **App Registrations**. You cannot use the SSO application you created in [Entra ID authentication](../../oidc) for SCIM. Its **Provisioning > Get started** button is disabled and Entra ID warns that out-of-the-box automatic provisioning is not supported. Create a separate non-gallery enterprise application for provisioning instead. Your existing SSO application registration continues to handle sign-in unchanged. ::: 1. In the Azure portal, open **Entra ID**, then **Enterprise applications**. 2. Select **New application**, then **Create your own application**. 3. Enter a name (for example, `Seqera SCIM provisioning`) and select **Integrate any other application you don't find in the gallery (Non-gallery)**. 4. Select **Create**. Use this application only for SCIM provisioning. Do not configure sign-on for it. ## Enable provisioning in Entra ID 1. In **Enterprise applications**, select the provisioning application you created in the previous section. 2. Open **Provisioning** and select **Get started**. 3. Set **Provisioning Mode** to **Automatic**. 4. Under **Admin Credentials**, provide: - **Tenant URL**: The Platform SCIM endpoint URL from [Get the Platform SCIM connection details](#get-the-platform-scim-connection-details). - **Secret Token**: The Platform bearer token from the same section. 5. Select **Test Connection**. Entra ID should report success. 6. Select **Save**. ## Scope and start provisioning :::caution Platform supports group provisioning only, and rejects every user create, update, and delete that Entra ID sends. Disable the **Provision Microsoft Entra ID Users** mapping (step 3) before you turn provisioning on. If you leave it enabled, every provisioning cycle fills the Entra ID provisioning log with failures and shows a provisioning error on the application in the Azure portal. Platform users are created automatically at SSO login and don't need SCIM. ::: 1. With **Provisioning** still open, expand **Settings**. 2. Set **Scope** to **Sync only assigned users and groups**. 3. Expand **Mappings** and select **Provision Microsoft Entra ID Users**. Under **Target Object Actions**, clear **Create**, **Update**, and **Delete**, then select **Save**. Leave the **Provision Microsoft Entra ID Groups** mapping enabled. 4. Save, then set **Provisioning Status** to **On**. 5. Return to the provisioning application's **Users and groups** tab and assign the groups you want Platform to receive. Entra ID runs an initial cycle within minutes and then syncs incrementally every ~40 minutes. ## Group display names vs. object IDs :::caution By default, Entra ID emits group **object GUIDs** in the `groups` claim, not display names. There are two options: - **Recommended**: Configure Entra ID to emit display names. In the application's **Token configuration**, add a **groups claim** and select **sAMAccountName** as the source where supported, or use a custom claims policy. This makes catalog entries and audit logs human-readable. - **Alternative**: Accept the default GUID emission. Use the GUID as the **IdP Group** value on each team. This works but makes the catalog harder to read. Pick one approach for your tenant and use it consistently. The GUID and the display name don't both flow at the same time. ::: ## Verify in Platform 1. In Platform, open **Organization settings > Group mapping**. 2. Select **Refresh**. The assigned Entra ID groups should appear in the catalog list after the first provisioning cycle. 3. The **Linked team** drop-down is now populated with the synced groups. If groups don't appear, open the **Provisioning logs** for the application in Entra ID and review any failed actions. ## Group rename and delete behavior Renames and deletes propagate automatically through SCIM: - **Rename**: The next provisioning cycle updates the catalog row's display name. Delegated teams that reference the group continue to work without interruption. - **Delete**: Entra ID issues a SCIM `DELETE` for the group, or removes the assignment from the enterprise application. Seqera removes the catalog row and synchronously purges members from any delegated team that referenced it. The affected teams remain in place with empty membership and an orphaned-team warning. ## Troubleshooting For SCIM provisioning issues, see [SCIM provisioning](../../../../../troubleshooting_and_faqs/authentication#scim-provisioning). --- ## SCIM provisioning with Okta Configure Okta to push your organization's groups to Platform over SCIM 2.0. Once provisioning is enabled, your Okta group directory appears in Seqera's IdP group catalog and stays in sync with renames, additions, and deletions automatically. :::info[**Prerequisites**]{#prerequisites} You will need the following to get started: - An Okta application configured as your Platform SSO connection. See [Okta authentication](../../oidc). - Organization owner access to your Platform organization. - Administrator access to your Okta tenant. ::: ## Get the Seqera SCIM connection details 1. In Platform, open **Organization settings > Group mapping**. 2. Copy the **SCIM endpoint URL**. It has the form `https:///api/orgs//scim/v2`. 3. Select **Generate token** to issue a SCIM bearer token. Copy it immediately. You can't view it again after closing the dialog. :::caution The bearer token grants write access to your group catalog. Store it in a secrets manager and rotate it on a schedule. To rotate, generate a new token in Seqera and update Okta's configuration. The previous token is revoked when the new token is issued. ::: ## Enable provisioning in Okta 1. Sign in to your Okta administrator console. 2. Open **Applications**, then select the application that fronts your Seqera SSO connection. 3. Open the **Provisioning** tab and select **Configure API integration**. 4. Select **Enable API integration** and provide: - **Base URL**: The Platform SCIM endpoint URL from the previous section, with `/Groups` removed (Okta appends the resource path). - **API token**: The Platform bearer token from the previous section. 5. Select **Test API Credentials**. Okta should report a successful connection. 6. Select **Save**. ## Enable group push 1. With the application still open, switch to the **Push Groups** tab. 2. Select **Push Groups**, then **Find groups by name** (or **By rule** for dynamic group sets). 3. Select the Okta groups you want available in Platform. 4. Confirm the push. Okta sends an initial provisioning batch. ## Verify in Platform 1. In Platform, open **Organization settings > Group mapping**. 2. Select **Refresh**. The pushed Okta groups should appear in the catalog list within a few seconds. 3. The **Linked team** drop-down is now populated with the synced groups. If groups don't appear, check the **Push Groups** status column in Okta for error details, and confirm that the **Provisioning** tab shows **Push Groups: ON**. ## Group rename and delete behavior Renames and deletes propagate automatically: - **Rename**: The next SCIM push updates the catalog row's display name. Delegated teams that reference the group continue to work without interruption. - **Delete**: Okta issues a SCIM `DELETE` for the group. Seqera removes the catalog row and synchronously purges members from any delegated team that referenced it. The affected teams remain in place with empty membership and an orphaned-team warning. ## Troubleshooting For SCIM provisioning issues, see [SCIM provisioning](../../../../../troubleshooting_and_faqs/authentication#scim-provisioning). --- ## Multi-organization routing Cloud Pro tokens carry an `org_id` claim that scopes every IdP delegation evaluation to a single organization. Enterprise SSO tokens don't carry such a claim. Platform routes by deployment topology and relies on a cross-organization uniqueness invariant on group display names. The rules on this page determine how Platform resolves a user's groups claim against each organization's catalog. ## Topology decision table | Topology | How users are routed | Group-name uniqueness | |----------|----------------------|------------------------| | **No SSO** | Not applicable. | Not applicable. | | **Single organization** | Trivially scoped to the single organization. | Not enforced; there is no second organization to collide with. | | **Multi-organization** | Evaluated against every organization the user is a member of. | **Enforced**. Group display names must be unique across all organizations on the instance. | ## The uniqueness invariant In a multi-organization Enterprise instance, when an administrator adds a group to the catalog (manually or through SCIM), Seqera checks the group's display name against every other organization's catalog on the instance. If another organization already has a row with the same display name, the operation fails: - **Manual add**: The form rejects the value with a `409 Conflict` and a message naming the conflicting organization. - **SCIM push**: Platform's SCIM endpoint returns `409 Conflict` for that group. The IdP's provisioning agent retries and surfaces the error in its administrator console. This is the mechanism that lets Seqera resolve a `groups` claim back to a specific organization's catalog at login. Without it, two organizations could both have a group called `engineering` and Platform couldn't determine which delegation rules to apply. :::info On Cloud Pro, the uniqueness check is skipped because the `org_id` claim disambiguates without it. ::: ## Resolving a conflict If a group-name conflict prevents you from adding a group: - Coordinate with the conflicting organization's owner to rename one of the groups in the upstream IdP. The rename propagates via SCIM (or is re-entered manually), and the catalog row becomes available again. - If renaming isn't possible, namespace the group in your IdP — for example, prefix the group with the organization's name (`acme-engineering` instead of `engineering`). There is no per-organization override. Uniqueness is enforced at the instance level. ## Cross-organization users When a user belongs to multiple organizations on the same instance, Platform evaluates their `groups` claim against every organization's delegated teams at login. Because group display names are unique instance-wide, each claim value maps unambiguously to one organization's catalog, and the user joins the matching teams in every organization where they apply. ## Operator guidance for new instances For new multi-organization Enterprise deployments, establish a naming convention for IdP groups before onboarding the first organization. Common patterns: - **Organization prefix**: `acme-eng-admins`, `beta-eng-admins`. Easy to read; explicit ownership. - **Reverse-DNS namespace**: `io.acme.eng.admins`. Compact; aligns with IdP best practice. - **Functional grouping with project codes**: `eng-NF-admins` where `NF` denotes a project. Useful when groups span organizations but require unique names. Document the convention in your organization onboarding checklist so administrators avoid `409 Conflict` errors when they configure delegation. --- ## IdP delegation overview IdP delegation lets you map a Seqera team to a group in your identity provider (IdP). After you delegate a team, the IdP becomes the sole authority for membership. Every time a user signs in through SSO, Seqera reads the `groups` claim from their token and updates the user's delegated-team memberships to match. IdP delegation requires a working OIDC SSO connection. To set up SSO before configuring delegation, see [Authentication](../overview). ## How it works Delegation has three components that you configure once per organization. ### The IdP group catalog Seqera maintains a per-organization catalog of IdP groups. The catalog populates the **IdP Group** drop-down on the group mapping page. Organization owners can select an IdP group when delegating a team. Groups appear in the catalog as soon as they're synced or entered, before any user has signed in. The catalog is populated in one of two ways: - **SCIM 2.0 push**: Your IdP pushes its group directory to Seqera's per-organization SCIM endpoint. Used with Okta and Entra ID. - **Manual entry**: For IdPs that don't support SCIM group sync (Google Workspace, Keycloak), an organization owner enters group identifiers in the catalog UI. A manually-entered group is automatically promoted to SCIM-managed if your IdP later pushes the same group. See [Manage your IdP group catalog](./group-catalog/overview). ### The `groups` claim At login, Seqera reads the user's IdP claims to decide which delegated teams they belong to. The `groups` claim must be present in the token and must contain the same group identifiers as your catalog. Unlike Cloud Pro, which authenticates through Auth0 and requires a connection-level mapping, Enterprise reads the IdP's tokens directly. Configure the claim at the IdP itself. See [IdP claim mapping](./claim-mapping). ### The Team's `IdP Group` field When an organization owner sets the **IdP Group** field on a team, the team becomes delegated. Delegation has the following effects: - The team is labeled **Managed in IdP** in the teams list. - The **Add member** and **Remove member** controls are hidden. - The team cannot be deleted until the **IdP groups** field is cleared. - The team's name, description, avatar, and workspace assignments remain editable. The same IdP group can only be assigned to a single team. Each team can reference exactly one IdP group. See [Delegate a team to an IdP group](../../../../orgs-and-teams/teams#delegate-a-team-to-an-idp-group). When a user logs in via SSO, Seqera evaluates their group claims and adds them to any delegated teams that match. The user must already exist in the Platform, but does not need to be a member of the organization that owns the team. :::info In deployments with more than one organization, a user does not need to be an existing member of an organization to be added to a delegated team in that organization. When their IdP group claim matches a delegated team, the user is added to both the team and its owning organization automatically. ::: ## What happens at login On every SSO login, Seqera evaluates each delegated team in your organization against the user's `groups` claim: - **Match found**: The user is added to the team if they aren't already a member, and gains the workspace access the team grants. - **No match and the user was previously a member**: The user is removed from the team, and the workspace access the team granted is revoked. - **No match and the user was never a delegation-driven member**: no change. Manual assignments to non-delegated teams are never touched by this evaluation. Users added manually to a team with no **IdP Group** value keep their membership regardless of their IdP claims. An **absent or empty** `groups` claim is treated as "member of no groups": the user is removed from every team they joined through delegation. Only a **malformed** `groups` claim (a value that is not a list) is ignored, leaving existing memberships unchanged. ## Multi-organization deployments Cloud Pro tokens carry an `org_id` claim that scopes evaluation to a single organization. Enterprise SSO tokens do not, so the platform routes by deployment topology and enforces a cross-organization uniqueness invariant on group display names. See [Multi-organization routing](./multi-org-routing) for the rules and conflict resolution. ## Audit trail Delegation activity is recorded in the [audit log](../../../../monitoring/audit-logs): - Setting, changing, or clearing the **IdP Group** field on a team produces a `team_updated` event with the previous and new value of `idpGroup`. - Each delegation-driven membership change at login produces a `team_member_added` or `team_member_removed` event. - Group catalog operations produce `idp_group_created`, `idp_group_updated`, and `idp_group_deleted` events so you can correlate catalog changes with downstream membership changes. - SCIM bearer token lifecycle operations produce `scim_token_created` and `scim_token_updated` events (the latter covers rotation and revocation), so changes to the token used against Seqera's SCIM endpoint are captured in the audit trail. SCIM-originated entries (operations performed by your IdP's provisioning agent against Seqera's SCIM endpoint) are attributed to a **System** operator rather than to a named administrator, because they authenticate with a SCIM bearer token. To correlate a SCIM event with a specific administrator action, match by `displayName` and timestamp against your IdP's provisioning logs. ## Set up delegation Complete these steps in order. Each step links to a dedicated guide. 1. [Configure authentication](../overview) for your Enterprise instance if you haven't already. 2. [Populate the IdP group catalog](./group-catalog/overview). Choose SCIM push or manual entry depending on your IdP. 3. [Configure the IdP to emit the `groups` claim](./claim-mapping) so it reaches Seqera at login. 4. If your instance hosts multiple organizations, review the [multi-organization routing rules](./multi-org-routing). 5. [Delegate a Team to an IdP group](../../../../orgs-and-teams/teams#delegate-a-team-to-an-idp-group). --- ## OpenID Connect Configure any OpenID Connect (OIDC) provider for single sign-on (SSO) to Seqera Platform. Keycloak, Microsoft Entra ID, and Okta are covered below, but the same steps apply to any OIDC-compliant provider. All OIDC providers use the same Seqera environment variables (`TOWER_OIDC_CLIENT`, `TOWER_OIDC_SECRET`, `TOWER_OIDC_ISSUER`) and the same callback URL. Only the application setup within the provider differs. :::note You can combine different OAuth and OIDC provider types. However, only one OIDC provider can be configured at a time. ::: ## Create an application with your identity provider Create an application (or client) in your identity provider, then note its client ID, client secret, and issuer URL for the next section. In your provider settings, set the callback address (also called the authorized redirect or sign-in redirect URI) to the following, replacing `` with your enterprise installation hostname (must be HTTPS): ``` https:///oauth/callback/oidc ``` :::info[Prerequisites]{#keycloak-prerequisites} You need the following: - A [Keycloak](https://www.keycloak.org/) instance - Admin access to create clients in Keycloak ::: See the [Keycloak documentation](https://www.keycloak.org/docs/latest/server_admin/#assembly-managing-clients_server_administration_guide) to configure Keycloak clients. **Create a Keycloak client** 1. In **Realm settings**, verify the **Endpoints** field includes _OpenID Endpoint Configuration_. 2. Go to **Clients** and select **Create**. 3. Configure the client with protocol `openid-connect`, access type `confidential`, and redirect URI `https:///oauth/callback/oidc`. 4. In the **Credentials** tab, note the **Secret**. 5. In the **Keys** tab, set **Use JWKS URL** to `OFF`. 6. Note the issuer URL from **Realm Settings > Endpoints > OpenID Configuration** (the `issuer` value in the JSON), e.g., `https://keycloak.example.com/auth/realms/master`. :::info[Prerequisites]{#entra-id-prerequisites} You need the following: - An Azure account with [Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/identity-platform/v2-protocols-oidc) access - Permission to create app registrations ::: See Microsoft's documentation on [registering an application](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app) for the registration steps. **Register an Entra ID application** 1. In the [Azure portal](https://portal.azure.com/), go to **Entra ID > App Registrations**. 2. Select **New Registration** and specify a name and supported account types. 3. Set the redirect URI to `https:///oauth/callback/oidc`. 4. Note the **Application (client) ID** from the app overview. 5. Go to **Certificates & secrets** and create a new client secret. Note the secret value. 6. Go to **Endpoints** and note the OpenID Connect metadata document URI (up to `v2.0`), e.g., `https://login.microsoftonline.com//v2.0`. **User consent settings** Configure user consent settings to **Allow user consent for apps** to ensure admin approval is not required for each login. See [User consent settings](https://learn.microsoft.com/en-us/azure/active-directory/manage-apps/configure-user-consent?pivots=portal#configure-user-consent-settings). :::note Compatibility note Users on Seqera Platform version 25.2.3 and below may need to set the following environment variable to resolve an authentication method incompatibility: ```env MICRONAUT_SECURITY_OAUTH2_CLIENTS_OIDC_OPENID_TOKEN_AUTH_METHOD=client_secret_post ``` ::: :::info[Prerequisites]{#okta-prerequisites} You need the following: - An [Okta](https://www.okta.com/) organization - Administrator access to create applications ::: See Okta's documentation on [creating OIDC app integrations](https://help.okta.com/en-us/content/topics/apps/apps_app_integration_wizard_oidc.htm) to set up the app. **Create an Okta app integration** 1. In the **Admin Console**, go to **Applications > Applications**. 2. Select **Create App Integration**. 3. Select **OIDC - OpenID Connect** as the sign-in method and **Web Application** as the application type. 4. Enter a name for the app, e.g., `Seqera`. 5. Set the sign-in redirect URI to `https:///oauth/callback/oidc`. 6. Set the sign-out redirect URI to `https:///logout`. 7. Note the **Client ID** and **Client secret** from the application settings. 8. Note the **Issuer** URL from **Sign On > OpenID Connect ID Token**. :::note Connection strings can differ based on the issuer type. Verify the issuer URL via the Okta console. ::: ## Configure Seqera Add the following environment variables to your Seqera backend service configuration: | Variable | Description | | :------- | :---------- | | `TOWER_OIDC_CLIENT` | The client ID provided by your identity provider | | `TOWER_OIDC_SECRET` | The client secret provided by your identity provider | | `TOWER_OIDC_ISSUER` | The issuer URL provided by your identity provider | | `TOWER_OIDC_SCOPES` | Optional. Comma-separated OAuth 2.0 scopes the OIDC client requests at login. Defaults to `openid,email,profile`. `openid` is always included. Add `groups` when your IdP only emits the group claim for a requested scope (see the note below). | Some providers require the full authentication service URL while others require only the SSO root domain (without the trailing sub-directories). Platform adds the `auth-oidc` Micronaut environment automatically when it detects `TOWER_OIDC_CLIENT` in the backend service environment at startup. The `cron` service never needs it. :::note Automatic detection reads operating system environment variables only, and runs before Platform loads values from a secret store. If you supply `TOWER_OIDC_CLIENT` from AWS Systems Manager Parameter Store or another store that Platform reads after startup, add `auth-oidc` to the `MICRONAUT_ENVIRONMENTS` environment variable on the backend service. Without it, Platform starts without the OIDC login option. ::: :::note If you plan to use IdP-delegated teams, your OIDC token must include a `groups` claim. Some providers (for example, Okta) only emit this claim when a matching scope is requested. In that case, add `groups` to the requested scopes with `TOWER_OIDC_SCOPES=openid,email,profile,groups`. See [IdP claim mapping](./idp-delegation/claim-mapping) for the per-IdP configuration steps. ::: ## Restrict access To restrict access to specific email addresses or domains, configure an allow list in `tower.yml`: ```yaml tower: auth: oidc: allow-list: - "*@your-company.example.com" - "specific-user@another-company.example.net" ``` See [User access allow list](./overview#user-access-allow-list) for more information. ## Platform as an OIDC provider The settings earlier on this page configure Seqera Platform as an OIDC *client* against your identity provider. The following settings apply when Platform *issues* OIDC tokens of its own. ### Additional static OIDC clients Seqera Platform ships with built-in static OIDC clients. To register further clients for a deployment-specific or enterprise integration, add them to `oauth-client.additional-static-clients` in `tower.yml`: ```yaml oauth-client: additional-static-clients: - client-id: my-integration # same client types and fields as the built-in static clients ``` Additional clients support the same client types and fields as the built-in `oauth-client.static-clients`, and Platform applies the same validation to both at startup. Platform searches both lists together for client lookup and conflict checks. A client ID that duplicates a built-in one fails startup with a `Duplicate client IDs across static and additional clients` error. Platform logs the validated client counts at startup. ### Access token audience Platform can stamp an `aud` claim on the OIDC access tokens it issues, and enforce that claim on incoming ones. | Environment variable | Configuration property | Description | | --- | --- | --- | | `TOWER_OIDC_ACCESS_TOKEN_AUDIENCE` | `tower.oidc.access-token.audience` | Audience stamped on issued OIDC access tokens. Defaults to the virtual audience `platform`, which covers every resource server behind the Platform API. | | `TOWER_OIDC_AUDIENCE_ENFORCEMENT_ENABLED` | `tower.oidc.audience-enforcement.enabled` | When `true`, Platform rejects its own OIDC access tokens whose `iss` or `aud` claims do not match. Default `false`. When `false`, Platform logs violations without rejecting tokens, for a warn-first rollout. | Personal access tokens are opaque and are unaffected by either setting. Enable enforcement only after confirming from the logs that your clients present the expected audience. :::warning When `TOWER_AUTH_TOKEN_SIGNING_RS256_ENABLED` is also set, the audience applies to the Platform token family as well, and Platform compares both claims against live configuration. Changing `TOWER_SERVER_URL` or `TOWER_OIDC_ACCESS_TOKEN_AUDIENCE` with enforcement on rejects every session, agent, and service token (not only OIDC ones) until they expire. ::: --- ## Authentication(Authentication) Seqera Platform supports email and various OAuth providers for login authentication. ## Identity providers Configure login authentication with any of the following identity providers: | Provider | Protocol | Configuration | | :------------------------------------------------------ | :--------- | :--------------- | | [Email](./email) | Magic link | `TOWER_SMTP_*` | | [GitHub](./github) | OAuth | `TOWER_GITHUB_*` | | [Google](./google) | OAuth | `TOWER_GOOGLE_*` | | [OpenID Connect](./oidc) (Keycloak, Entra ID, Okta, …) | OIDC | `TOWER_OIDC_*` | ## OpenID Connect configuration :::note You can combine different OAuth and OIDC provider types. However, only one OIDC provider can be configured at a time. ::: For OIDC providers, configure authentication with these environment variables: | Variable | Description | | :------------------ | :------------------------------------------------------------------------------------------ | | `TOWER_OIDC_CLIENT` | The client ID provided by your authentication service | | `TOWER_OIDC_SECRET` | The client secret provided by your authentication service | | `TOWER_OIDC_ISSUER` | The authentication service URL to which Seqera connects to authenticate the sign-in request | Some providers require the full authentication service URL while others require only the SSO root domain (without the trailing sub-directories). In your OpenID provider settings, specify the following URL as a callback address or authorized redirect: ``` https:///oauth/callback/oidc ``` :::note If you plan to use IdP-delegated teams, your OIDC token must include a `groups` claim. See [IdP claim mapping](./idp-delegation/claim-mapping) for the per-IdP configuration steps. ::: ## Root users Root users have administrative access to all Platform resources. Configure root users by their user ID or email address in a comma-separated list: **Environment variable** ```env TOWER_ROOT_USERS=1,admin@your-company.example.com ``` **tower.yml** ```yaml tower: admin: root-users: "1,admin@your-company.example.com" ``` ## JWT secret Configure the secret key used to sign JWT tokens for user authentication sessions. This is a required security setting for all Platform deployments. :::warning The JWT secret must remain consistent across all backend instances and restarts. Changing this value will invalidate all active user sessions and log out all users. ::: **Environment variable** ```env TOWER_JWT_SECRET= ``` **Requirements:** - Minimum 35 characters recommended - Use a cryptographically secure random string - Keep this value secret and do not commit to version control **Generate a secure value:** ```bash openssl rand -base64 48 ``` This secret is used to sign both access tokens and refresh tokens for user sessions. ## Disable email login Disable email-based (magic link) authentication when OAuth providers are configured. :::note This setting only takes effect when at least one OAuth provider (GitHub, Google) or OIDC is configured. ::: **Environment variable** ```env TOWER_AUTH_DISABLE_EMAIL=true ``` **tower.yml** ```yaml tower: auth: disable-email: true ``` ## Session management Platform login sessions remain active as long as the application browser window remains open and active. Sessions use short-lived access tokens that are automatically refreshed via heartbeat. | Setting | Default | Description | |:------------------------------------------------------------------|:-----------|:----------------------------------------------------------------------------| | `micronaut.security.token.generator.access-token.expiration` | 3600s (1h) | Short-lived token, auto-refreshed via heartbeat | | `micronaut.security.token.jwt.generator.refresh-token.expiration` | 6h | Session idle timeout — users are logged out after this period of inactivity | | `micronaut.security.token.refresh.cookie.cookie-max-age` | 12h | Browser cookie lifetime (should be ≥ refresh token) | **tower.yml** ```yaml micronaut: security: token: jwt: generator: refresh-token: expiration: 8h generator: access-token: expiration: 3600 refresh: cookie: cookie-max-age: 10h ``` ## User access allow list Restrict access to specific user email addresses or domains. Allow list entries are case-insensitive. Replace `` with `github`, `google`, or `oidc`. Use `oidc` for any authentication service based on OpenID Connect (Okta, Entra ID, Keycloak, etc.). Include each provider separately if you configure more than one. **tower.yml** ```yaml tower: auth: : allow-list: - "*@your-company.example.com" - "specific-user@another-company.example.net" ``` ## IdP delegation and group claims Seqera Platform Enterprise supports IdP-delegated teams: organization owners can map a Seqera team to an IdP group, after which the IdP becomes the sole authority for who belongs to that team. Memberships are evaluated on every SSO login. :::note IdP claims mapping is **enabled by default** on Enterprise from version 26.1.4. To turn it off, set `TOWER_IDP_CLAIMS_MAPPING_ENABLED=false`. On earlier versions the feature is disabled by default and must be enabled explicitly. ::: The feature is gated per organization: it applies only to organizations with an active SSO connection, and — when the allowlist below is configured — only to the organizations on that list. **Environment variables** | Variable | Description | | :-- | :-- | | `TOWER_IDP_CLAIMS_MAPPING_ENABLED` | Master switch for IdP claims mapping. Defaults to `true` on Enterprise (26.1.4 and later). Set to `false` to disable the feature for the whole instance. | | `TOWER_IDP_CLAIMS_MAPPING_ALLOWED_ORGANIZATIONS` | Optional comma-separated list of organization IDs allowed to use the feature. When unset, all organizations that pass the other checks are allowed; when set but empty, no organization is allowed; when non-empty, only the listed organizations are allowed. | **tower.yml** ```yaml tower: idp-claims-mapping: enabled: true # Optional allowlist; omit to allow all SSO-enabled organizations allowed-organizations: "100,200" ``` For delegation to work, your IdP must: - Push or expose its group directory to Seqera. See [Manage your IdP group catalog](./idp-delegation/group-catalog/overview) for the SCIM 2.0 push and manual-entry options. - Include a `groups` claim in the tokens it issues. See [IdP claim mapping](./idp-delegation/claim-mapping) for protocol-specific guidance. Once those two pieces are in place, see [IdP delegation overview](./idp-delegation/overview) for the runtime model and [Delegate a Team to an IdP group](../../../orgs-and-teams/teams#delegate-a-team-to-an-idp-group) for the administrator procedure. If your Enterprise instance hosts more than one organization, review the [multi-organization routing rules](./idp-delegation/multi-org-routing) before configuring delegation. --- ## AWS Parameter Store From version 23.1, Seqera Platform Enterprise can fetch configuration values from the AWS Parameter Store. :::caution `TOWER_DB_USER`, `TOWER_DB_PASSWORD`, and `TOWER_DB_URL` values must be specified using **environment variables** during initial Seqera Enterprise deployment in a new environment. A new installation will fail if DB values are only defined in `tower.yml` or the AWS Parameter Store. After the database has been created, these values can be added to AWS Parameter Store entries and removed from your environment variables. ::: ## Configuration values not supported in AWS Parameter Store Due to the order of operations when deploying Seqera Enterprise, some configuration values can only be retrieved from **environment variables** (`tower.env`). The following configuration values are not supported by AWS Parameter Store and must be set as environment variables: | Environment Variable | Description | Value | | ------------------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | | `TOWER_DB_USER` | The user account to access your database. If you are using an external database, you must create this user manually. **For installation in a new environment, this value must be set as an environment variable.** | Default: `tower` | | `TOWER_DB_PASSWORD` | The user password to access your database. If you are using an external database, you must create this password manually. **For installation in a new environment, this value must be set as an environment variable.** | Default: `tower` | | `TOWER_DB_URL` | The URL to access your database. **For installation in a new environment, this value must be set as an environment variable.** | Example: `jdbc:mysql://db:3306/tower?permitMysqlScheme=true` | | `TOWER_APP_NAME` | Application name. To run multiple instances of the same Seqera account, each instance must have a unique name, e.g., `tower-dev` and `tower-prod`. **Can also be set in `tower.yml` with `tower.appName`.** | Default: `tower` | | `TOWER_ENABLE_AWS_SES` | Set `true` to enable AWS Simple Email Service for sending Seqera emails instead of SMTP. | Default: `false` | | `TOWER_ENABLE_PLATFORMS` | A comma-separated list of execution backends to enable. **At least one is required.** | `altair-platform,awsbatch-platform,awscloud-platform,azbatch-platform,eks-platform,googlebatch-platform,googlecloud-platform,gke-platform,k8s-platform,local-platform,lsf-platform,moab-platform,slurm-platform` | | `TOWER_ENABLE_UNSAFE_MODE` | Set to `true` to allow HTTP connections to Seqera. HTTP must not be used in production deployments. HTTPS is used by default from version 22.1.x. | Default: `false` | ## Configure Seqera to use AWS Parameter Store values To enable Seqera use AWS Parameter Store values: 1. Grant [AWS Parameter Store permissions](https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-paramstore-access.html) to your Seqera host instance. 2. Add `TOWER_ENABLE_AWS_SSM=true` in the `tower.env` configuration file. 3. Create individual parameters in the AWS Parameter Store (see below). 4. Start your Seqera instance and confirm the following entries appear in the **backend** container log: ```bash [main] - INFO i.m.context.DefaultBeanContext - Reading bootstrap environment configuration [main] - INFO i.m.d.c.c.DistributedPropertySourceLocator - Resolved 2 configuration sources from client: compositeConfigurationClient(AWS Parameter Store) ``` ## Create configuration values in AWS Parameter Store Store Seqera configuration values as individual parameters in the AWS Parameter Store. :::caution The default application name is `tower-app`. To deploy multiple instances from the same Seqera Enterprise account, set a custom application name for each instance with the `micronaut.application.name` value in your `tower.yml` configuration file. ::: We recommend storing sensitive values, such as database passwords, as SecureString-type parameters. These parameters require additional IAM KMS key permissions to be decrypted. Seqera does not support StringList parameters. Configuration parameters with multiple values can be created as comma-separated lists of String type. To create Seqera configuration parameters in AWS Parameter Store, do the following: 1. Navigate to the **Parameter Store** from the **AWS Systems Manager Service** console. 2. From the **My parameters** tab, select **Create parameter** and populate as follows: | Field | Description | | ----- | ----------- | | **Name** | Use the format `/config//`. `` follows the `tower.yml` nesting hierarchy. See the [configuration overview](./overview) for specific paths.**Example: `/config/tower-app/mail.smtp.password : `** | | **Description** | (Optional) Description for the parameter. | | **Tier** | Select **Standard**. | | **Type** | Use **SecureString** for sensitive values like passwords and tokens. Use **String** for everything else. | | **Data type** | Select **text**. | | **Value** | Enter a plain text value (this is the configuration value used in Seqera). | --- ## Mirroring container images Mirroring Seqera container images to your own registry is recommended for production deployments. This ensures your deployments are not impacted by external registry availability and supports air-gapped environments. ## Registry-native replication Use your container registry's built-in replication features to automatically sync images from `cr.seqera.io`: - [Amazon ECR replication](https://docs.aws.amazon.com/AmazonECR/latest/userguide/replication.html) - [Azure ACR artifact caching](https://learn.microsoft.com/en-us/azure/container-registry/artifact-cache-overview) - [Harbor replication](https://goharbor.io/docs/latest/administration/configuring-replication/) ## Skopeo Use [Skopeo](https://github.com/containers/skopeo) (v1.15+) when your registry doesn't support native replication. Authenticate with the Seqera registry: ```bash skopeo login --username 'robot$private+YOUR_ROBOT_USERNAME' -p 'YOUR_PASSWORD' cr.seqera.io ``` Create a YAML file (`seqera-images.yaml`) to specify which images to sync: ```yaml cr.seqera.io: images-by-semver: enterprise/platform/backend: ">= v26.2.2" enterprise/platform/frontend: ">= v26.2.2" enterprise/platform/migrate-db: ">= v26.2.2" # Co-Scientist MCP server, if you deploy Co-Scientist enterprise/mcp/server: ">= 1.4.3" ``` Run the sync: ```bash skopeo sync --scoped --src yaml --dest docker seqera-images.yaml YOUR_REGISTRY ``` Schedule periodic sync jobs to keep images current. See the [Skopeo sync documentation](https://github.com/containers/skopeo/blob/main/docs/skopeo-sync.1.md) for advanced usage. --- ## Networking ## HTTP proxy environment variables :::caution Proxies that require passwords aren't supported. ::: If your Seqera Platform Enterprise instance must access the internet via a proxy server, configure the following case-insensitive environment variables: - `http_proxy`: The proxy server for HTTP connections. - `https_proxy`: The proxy server for HTTPS connections. - `no_proxy`: One or more host names that bypass the proxy server. In the following example, `alice.example.com:8080` is configured as a proxy for all HTTP and HTTPS traffic, except for traffic to the `internal.example.com` and `internal2.example.com` hosts. ```env export http_proxy='alice.example.com:8080' export https_proxy='alice.example.com:8080' export no_proxy=internal.example.com,internal2.example.com ``` ## Isolated environments If you're deploying Seqera in an environment that has no external internet access, ensure that no pipeline assets or parameters in your configuration contain external links, as this will lead to connection failures. ## Mail proxy server Mail proxy server configuration details must be set either in `tower.yml` or AWS Parameter Store. **tower.yml** ::table{file=configtables/mail_server_proxy_yml.yml} **AWS Parameter Store** ::table{file=configtables/mail_server_proxy_aws.yml} --- ## Configuration :::note Nextflow Tower Enterprise is now Seqera Platform Enterprise. Existing configuration parameters, configuration files, and API endpoints that include _Tower_ currently remain unchanged. ::: Set Seqera configuration values using environment variables, a `tower.yml` configuration file, or individual values stored in AWS Parameter Store. Sensitive values such as database passwords should be stored securely (e.g., as SecureString type parameters in AWS Parameter Store). Declare environment variables in a [tower.env](../_templates/docker/tower.env) file. For example: ```bash TOWER_CONTACT_EMAIL=hello@foo.com TOWER_SMTP_HOST=your.smtphost.com ``` See the `Environment variables` option in each section below. Declare YAML configuration values in a [tower.yml](../_templates/docker/tower.yml) file. For example: ```yml mail: from: "hello@foo.com" smtp: host: "your.smtphost.com" ``` See the `tower.yml` option in each section below. YAML configuration keys on this page are listed in "dot" notation, i.e., the SMTP host value in the snippet above is represented as `mail.smtp.host` in the tables that follow. Don't declare duplicate keys in your `tower.yml` configuration file. Platform will only enforce the last instance of configuration keys that are defined more than once, for example: ```yaml # This block will not be enforced due to the duplicate `tower` key below tower: trustedEmails: - user@example.com # This block will be enforced because it's defined last tower: auth: oidc: - "*@foo.com" ``` In a [Helm chart](../platform-helm) deployment, the chart writes the `platform.YAMLConfigFileContent` field of your values file to the `tower.yml` file that Platform reads. Reserve this field for structured values that no environment variable can express, such as the trusted email list or a [custom navigation menu](#custom-navigation-menu): ```yaml platform: YAMLConfigFileContent: |- tower: trustedEmails: - "*@foo.com" ``` Where the chart exposes a dedicated value for a setting, such as `platform.contactEmail` or `platform.smtp.host`, use that value instead. For other settings that accept an environment variable, use `backend.extraEnvVars` and `cron.extraEnvVars`. These apply per deployment, so set both where the backend and cron pods need the same value. AWS Parameter Store configuration is only supported for AWS deployments. Create parameters in the AWS Parameter Store individually, using the format `/config// : `. For example: ```bash /config/tower-app/mail.smtp.user : /config/tower-app/mail.smtp.password : ``` :::caution The default application name is `tower-app`. To deploy multiple instances from the same Seqera Enterprise account, set a custom application name for each instance with the `micronaut.application.name` value in your `tower.yml` configuration file. ::: Sensitive values (such as database passwords) should be SecureString type parameters. See [AWS Parameter Store](./aws_parameter_store) for detailed instructions. ## Configuration values not supported in tower.yml or AWS Parameter Store Due to the order of operations when deploying Seqera Enterprise, some configuration values can only be retrieved from **environment variables** (`tower.env`). The following configuration values are not supported for `tower.yml` or AWS Parameter Store configuration and must be set as environment variables: ::table{file=configtables/req_env_vars.yml} ## Basic configuration Basic configuration options such as the Seqera instance server URL, application name, and license key. ::table{file=configtables/generic_config_env.yml} YAML configuration keys in this table are listed in "dot" notation, i.e., a nested value: ```yaml ... mail: smtp: host: "your.smtphost.com" ... ``` is represented as `mail.smtp.host`. ::table{file=configtables/generic_config_yml.yml} AWS Parameter Store configuration is only supported for AWS deployments. Replace `{prefix}` in each configuration path with `/config/`, where `application_name` is `tower` or your custom application name. See [AWS Parameter Store](./aws_parameter_store). ::table{file=configtables/generic_config_aws.yml} ## Seqera and Redis databases Configuration values that control Seqera's interaction with databases and Redis instances. `TOWER_DB_USER`, `TOWER_DB_PASSWORD`, and `TOWER_DB_URL` must be specified using environment variables during initial Seqera Enterprise deployment in a new environment. A new installation will fail if DB values are only defined in `tower.yml` or the AWS Parameter Store. Once the database has been created, these values can be added to `tower.yml` or [AWS Parameter Store](./aws_parameter_store) entries and removed from your environment variables. :::note **Database and cache version requirements:** From Seqera Enterprise version 26.1: - MySQL 8.4, the long-term support (LTS) release, is the recommended and tested database version. MySQL 5.7 and 8.0 have reached upstream end-of-life and are no longer tested or supported. Migrate to MySQL 8.4 before you upgrade. - Seqera Enterprise supports Redis 7.2, Redis 7.4, and Valkey 7.x. Redis 6.x is not supported. For the full support matrices, see [Database changes](../upgrade#database-changes) and [Cache layer changes](../upgrade#cache-layer-changes-redis-eol-and-valkey-support). Follow your cloud provider specifications to upgrade your instance. ::: If you use a database **other than** the provided `db` container, you must create a user and database schema manually. ```SQL CREATE DATABASE tower; ALTER DATABASE tower CHARACTER SET utf8 COLLATE utf8_bin; CREATE USER 'tower' IDENTIFIED BY ; GRANT ALL PRIVILEGES ON tower.* TO tower@'%' ; ``` ```SQL GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER on tower.* TO tower@'%'; ``` ### Managed Redis services Seqera supports managed Redis services such as [Amazon ElastiCache](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/WhatIs.html), [Azure Managed Redis](https://learn.microsoft.com/azure/redis/overview), or [Google Memorystore](https://cloud.google.com/memorystore/docs/redis). :::caution Microsoft is retiring Azure Cache for Redis. As of April 1, 2026, new customers cannot create instances, and from October 1, 2026, no new instances can be created. For new Azure deployments, use [Azure Managed Redis](https://learn.microsoft.com/azure/redis/overview). ::: When using a managed Redis service, you must specify the service IP address or DNS name for the `TOWER_REDIS_URL` as described in the following sections. - Use a single-node cluster, as multi-node clusters are not supported - Use an instance with at least 6 GB capacity ([cache.m4.large](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/CacheNodes.SupportedTypes.html) or greater) - Specify your private ElastiCache instance in the Seqera environment variables: ```bash TOWER_REDIS_URL=redis://:6379 ``` - Use a single-node cluster, as multi-node clusters are not supported - Choose a production-capable [Azure Managed Redis tier](https://learn.microsoft.com/azure/redis/overview) appropriate for your workload - Specify your private Azure Managed Redis instance in the Seqera environment variables: ```bash TOWER_REDIS_URL=redis://:6379 ``` - Use a single-node cluster, as multi-node clusters are not supported - Use an instance with at least 6 GB capacity ([M2](https://cloud.google.com/memorystore/docs/redis/pricing#instance_pricing) or greater) - Specify your private Memorystore instance in the Seqera environment variables: ```bash TOWER_REDIS_URL=redis://:6379 ``` If you run the Redis service as a container in your Docker or Kubernetes installation, specify the service name as part of the `TOWER_REDIS_URL`: ```bash TOWER_REDIS_URL=redis://redis:6379 ``` ### Database and Redis manual configuration If the DB username and password variables are left empty when using [Docker Compose](../platform-docker-compose), default `tower` database values are applied automatically. With [Kubernetes](../platform-kubernetes) and custom DB deployments, `tower` values are not pre-filled. :::note We recommend using managed cloud database services for production deployments. ::: ::table{file=configtables/db_env.yml} `TOWER_DB_USER`, `TOWER_DB_PASSWORD`, and `TOWER_DB_URL` must be specified using **environment variables** during initial Seqera Enterprise deployment in a new environment. YAML configuration keys in this table are listed in "dot" notation, i.e., a nested value: ```yaml ... mail: smtp: host: "your.smtphost.com" ... ``` is represented as `mail.smtp.host`. ::table{file=configtables/db_yml.yml} AWS Parameter Store configuration is only supported for AWS deployments. `TOWER_DB_USER`, `TOWER_DB_PASSWORD`, and `TOWER_DB_URL` must be specified using **environment variables** during initial Seqera Enterprise deployment in a new environment. Replace `{prefix}` in each configuration path with `/config/`, where `application_name` is `tower` or your custom application name. See [AWS Parameter Store](./aws_parameter_store). ::table{file=configtables/db_aws.yml} ## Opt-in Seqera features Configuration values that enable opt-in Seqera features per instance or workspace. ### Core features ::table{file=configtables/features_env.yml} ### Data features Configuration values used by Seqera for Datasets, Data Explorer, Data Lineage, and Studios. ::table{file=configtables/data_features_env.yml} ::table{file=configtables/data_features_yml.yml} ### Co-Scientist Configuration values Seqera Platform uses to reach a [Co-Scientist](../install-seqera-coscientist.mdx) deployment, control the Co-Scientist panel and agents, and restrict the Projects view. Set these only when Co-Scientist is deployed alongside your installation, except `TOWER_SCIENTIST_VIEW_ALLOWED_WORKSPACES`, which applies either way. ::table{file=configtables/co_scientist_env.yml} :::note Platform fails to start if `TOWER_AGENT_BACKEND_URL` is not a valid URI, does not use `https`, has no host, or contains a query string or fragment. ::: ## Cryptographic options Configuration values used by Seqera to encrypt your data. :::caution Do not modify your crypto secret key between starts. Changing this value will prevent the decryption of existing data. ::: ::table{file=configtables/crypto_env.yml} YAML configuration keys in this table are listed in "dot" notation, i.e., a nested value: ```yaml ... mail: smtp: host: "your.smtphost.com" ... ``` is represented as `mail.smtp.host`. ::table{file=configtables/crypto_yml.yml} AWS Parameter Store configuration is only supported for AWS deployments. Replace `{prefix}` in each configuration path with `/config/`, where `application_name` is `tower` or your custom application name. See [AWS Parameter Store](./aws_parameter_store). ::table{file=configtables/crypto_aws.yml} ### Secret key rotation Rotate the key used to encrypt the credentials and secrets stored in your Platform database. Encryption key rotation is a security best practice and should be performed at an interval specified by your organization's security requirements, or in the event of a suspected compromise of your secret key. Enable rotation by setting the following configuration values: - `TOWER_SECRET_ROTATION_ENABLED=true` - `TOWER_SECRET_ROTATION_PREVIOUS_KEY=` - `TOWER_CRYPTO_SECRETKEY=` - `tower.secret.rotation.enabled: true` - `tower.secret.rotation.previous-key: ` - `tower.crypto.secretKey: ` - `/tower/secret-rotation/enabled: true` - `/tower/secret-rotation/previous-key: ` - `/tower/crypto/secretKey: ` With rotation enabled and the previous and new key values set, secret key rotation will run as part of the Platform cron service during application startup. Normal application startup is not affected by this process, and Platform is fully operational while the credentials and secrets in your database are being encrypted using your new secret key. :::warning - To prevent data loss, perform a backup of your Platform database and securely back up your current crypto secret key before enabling and performing key rotation. - All backend pods or containers for your Enterprise deployment must contain the same previous and new secret key values in their Platform config and must be in a ready/running state before starting the Platform cron service. ::: The [Admin panel](../../administration/overview.md#encryption) **Encryption** tab displays the status of completed or errored encryption tasks. ## Backend memory requirements The Platform backend and cron services run on the Java Virtual Machine (JVM). Allocate at least 4 GB of memory to each service for stable operation under load. For Kubernetes, set resource limits in your pod specifications: ```yaml resources: limits: memory: "4Gi" requests: memory: "4Gi" ``` For Docker Compose, set memory limits in your service definitions: ```yaml services: backend: mem_limit: 4g memswap_limit: 4g ``` :::note These default memory allocation limits are included in the Kubernetes manifest templates ([tower-svc.yml](../_templates/k8s/tower-svc.yml) and [tower-cron.yml](../_templates/k8s/tower-cron.yml)). For Docker Compose, add the `mem_limit` settings to your service definitions as shown above. ::: ### JVM memory tuning For production deployments, configure JVM memory parameters with the `JAVA_OPTS` environment variable. This baseline configuration suits most deployments: ```bash JAVA_OPTS="-Xms1000M -Xmx2000M -XX:MaxDirectMemorySize=800m -Dio.netty.maxDirectMemory=0 -Djdk.nio.maxCachedBufferSize=262144" ``` The `JAVA_TOOL_OPTIONS` environment variable is a supported alternative to `JAVA_OPTS`. The JVM reads it directly at startup and confirms pickup with a `Picked up JAVA_TOOL_OPTIONS` line in the service logs. Set one variable or the other, not both. Options passed on the command line (which is how `JAVA_OPTS` is applied) take precedence over `JAVA_TOOL_OPTIONS` when the same flag appears in both. :::note These default JVM memory settings are included in the configuration templates provided in these docs: - Kubernetes: [tower-svc.yml](../_templates/k8s/tower-svc.yml) and [tower-cron.yml](../_templates/k8s/tower-cron.yml) - Docker Compose: [tower.env](../_templates/docker/tower.env) ::: | Parameter | Description | | --- | --- | | `-Xms` / `-Xmx` | Initial and maximum heap size — the memory pool for Java objects. | | `-XX:MaxDirectMemorySize` | Off-heap memory for NIO operations, network buffers, and file I/O. Handles concurrent workflow API operations. | | `-Dio.netty.maxDirectMemory=0` | Disables Netty's internal memory tracking and relies on the JVM direct memory limit instead. | | `-Djdk.nio.maxCachedBufferSize` | Limits the size of cached NIO buffers to prevent excessive memory retention. | Adjust these baseline values based on the symptoms you observe. Increase `-XX:MaxDirectMemorySize` if you observe: - `OutOfMemoryError: Direct buffer memory` in the logs - High concurrent workflow launch rates (more than 100 simultaneous workflows) - Large configuration payloads or heavy API usage Increase heap memory (`-Xmx`) if you observe: - `OutOfMemoryError: Java heap space` in the logs - Garbage collection pauses that affect performance - Growing memory usage under sustained load For deployments running 200 or more concurrent workflows, increase the heap and direct memory limits: ```bash JAVA_OPTS="-Xms1000M -Xmx3000M -XX:MaxDirectMemorySize=1600m -Dio.netty.maxDirectMemory=0 -Djdk.nio.maxCachedBufferSize=262144" ``` Set the container or pod memory limit higher than the JVM limits to accommodate non-heap memory usage. :::warning These are starting values. Monitor your deployment's memory usage and adjust for your workload. Undersized memory allocation can cause out-of-memory (OOM) failures and service instability. ::: ## Compute environments Configuration values to enable computing platforms and customize Batch Forge resource naming. ::table{file=configtables/compute_env.yml} ### Available platform values `TOWER_ENABLE_PLATFORMS` (`tower_enable_platforms` in the Terraform installer) controls which compute environment types are available in your installation. A platform must be listed here before a compute environment of that type can be created — if a compute environment type does not appear in the Platform UI, the corresponding value is missing from this variable. | Platform value | Compute environment type | | :------------- | :----------------------- | | `awsbatch-platform` | [AWS Batch](../../compute-envs/aws-batch) | | `awscloud-platform` | [AWS Cloud](../../compute-envs/aws-cloud) | | `azbatch-platform` | [Azure Batch](../../compute-envs/azure-batch) | | `azcloud-platform` | [Azure Cloud](../../compute-envs/azure-cloud) | | `googlebatch-platform` | [Google Cloud Batch](../../compute-envs/google-cloud-batch) | | `googlecloud-platform` | [Google Cloud](../../compute-envs/google-cloud) | | `eks-platform` | [Amazon EKS](../../compute-envs/eks) | | `gke-platform` | [Google Kubernetes Engine](../../compute-envs/gke) | | `k8s-platform` | [Kubernetes](../../compute-envs/k8s) | | `slurm-platform` | [Slurm](../../compute-envs/hpc) | | `lsf-platform` | [IBM Spectrum LSF](../../compute-envs/hpc) | | `uge-platform` | [Grid Engine](../../compute-envs/hpc) | | `altair-platform` | [Altair PBS Pro](../../compute-envs/hpc) | | `moab-platform` | [Moab](../../compute-envs/hpc) | | `local-platform` | Local (evaluation and development only) | Specify values as a comma-separated list with no spaces: ```bash TOWER_ENABLE_PLATFORMS=awsbatch-platform,azcloud-platform,k8s-platform ``` :::note Azure Batch and Azure Cloud are separate compute environment types with separate platform values. Enabling `azbatch-platform` does not make Azure Cloud compute environments available, and vice versa. ::: ### Compute environment cleanup A scheduled cron job can transition compute environments that are stuck in `CREATING` or `DELETING` states into terminal states (`ERRORED` or `INVALID`). The cleanup job is disabled by default. ::table{file=configtables/compute_env_cleanup_env.yml} ### Workspace orphan cleanup Two scheduled cron jobs can soft-delete credentials and compute environments whose workspace has been deleted. Both jobs are disabled by default. ::table{file=configtables/workspace_orphan_cleanup_env.yml} ## Git integration Seqera Platform has built-in support for public and private Git repositories. Create [Git provider credentials](../../git/overview) to allow Seqera to interact with the following services: - [GitHub](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) - [BitBucket](https://confluence.atlassian.com/bitbucketserver/personal-access-tokens-939515499.html) - [GitLab](https://gitlab.com/profile/personal_access_tokens) - [Gitea](https://docs.gitea.io/en-us/development/api-usage/#generating-and-listing-api-tokens) - [Azure Repos](https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate) :::caution Credentials configured in your SCM providers list override Git credentials in your (organization or personal) workspace. ::: Public Git repositories can be accessed without authentication, but are often subject to [throttling](https://docs.github.com/en/rest/overview/resources-in-the-rest-api?apiVersion=2022-11-28#rate-limits-for-requests-from-personal-accounts). We recommend always adding Git credentials to your Seqera workspace, regardless of the repository type you use. Credentials and other secrets must not be hard-coded in environment variables in production environments. Credentials added using the application UI are SHA256-encrypted before secure storage and not exposed by any Seqera API. ::table{file=configtables/git_env.yml} Credentials and other secrets must not be stored in plain text in production environments. Credentials added using the application UI are SHA256-encrypted before secure storage and not exposed by any Seqera API. YAML configuration keys in this table are listed in "dot" notation, i.e., a nested value: ```yaml ... mail: smtp: host: "your.smtphost.com" ... ``` is represented as `mail.smtp.host`. ::table{file=configtables/git_yml.yml} AWS Parameter Store configuration is only supported for AWS deployments. Replace `{prefix}` in each configuration path with `/config/`, where `application_name` is `tower` or your custom application name. See [AWS Parameter Store](./aws_parameter_store). ::table{file=configtables/git_aws.yml} ### Local repositories Seqera Enterprise can connect to workflows stored in local Git repositories. To do so, volume mount your local repository folder in your Seqera backend container. Then, update your `tower.yml`: ```yml tower: pipeline: allow-local-repos: - /path/to/repo ``` ## Mail server Configure values for SMTP email service integration. Production SMTP hosts must use a TLS-protected connection. See [SSL/TLS](../configuration/ssl_tls). AWS deployments also support [Amazon Simple Email Service (SES)](https://aws.amazon.com/ses/). ### SMTP service integration To use an SMTP gateway for mail service, set SMTP user and password values to `null`. :::caution Your organization's email security policy may prevent the `TOWER_CONTACT_EMAIL` address from receiving Seqera emails. If this occurs after successful SMTP configuration, you may need to configure `spf`, `dkim`, and `dmarc` records for your domain. Contact your IT support staff for further assistance. ::: ::table{file=configtables/mail_server_env.yml} YAML configuration keys in this table are listed in "dot" notation, i.e., a nested value: ```yaml ... mail: smtp: host: "your.smtphost.com" ... ``` is represented as `mail.smtp.host`. ::table{file=configtables/mail_server_yml.yml} AWS Parameter Store configuration is only supported for AWS deployments. Replace `{prefix}` in each configuration path with `/config/`, where `application_name` is `tower` or your custom application name. See [AWS Parameter Store](./aws_parameter_store). ::table{file=configtables/mail_server_aws.yml} ### AWS SES integration In AWS deployments, you can use AWS Simple Email Service (SES) instead of traditional SMTP for sending Seqera platform emails. :::note Simple Email Service (SES) is only supported in Seqera deployments on AWS. ::: To configure AWS SES as your Seqera email service: 1. Set `TOWER_ENABLE_AWS_SES=true` in your environment variables. 2. Specify the email address used to send Seqera emails with one of the following: - the `TOWER_CONTACT_EMAIL` environment variable - a `mail.from` entry in `tower.yml` - a `/config//mail/from` AWS Parameter Store entry 3. The [AWS SES service](https://docs.aws.amazon.com/ses/index.html) must run in the same region as your Seqera instance. 4. The [Seqera IAM role](../../compute-envs/aws-batch#iam-user-creation) must include the `ses:SendRawEmail` permission. ## Nextflow launch container :::caution Do not replace the [Seqera-provided default image](../../functionality_matrix/overview) unless absolutely necessary. ::: | Environment Variable | Description | Value | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | | `TOWER_LAUNCH_CONTAINER` | The container image to run the Nextflow execution. This setting overrides the launch container selection for all organizations and workspaces in your account, and disables the per-run [Nextflow version](../../launch/advanced#nextflow-version) selector. | Example: `quay.io/seqeralabs/nf-launcher:j17-23.04.3` | ### Component compatibility catalog The component compatibility catalog records which Nextflow, Fusion, and `nf-launcher` versions work with this Seqera Platform version. The Nextflow version selector and launch validation read from it. ::table{file=configtables/catalog_env.yml} ## Seqera API Enable the API endpoints to host the Seqera Enterprise OpenAPI specification and use the [tw CLI](https://github.com/seqeralabs/tower-cli). Set custom API rate limits and timeouts. :::note To configure API rate limit environment variables, you must add `ratelim` to the `MICRONAUT_ENVIRONMENTS`. Without `ratelim` being set, the rate limit configuration variables below are ignored. ::: | Environment variable | Description | Value | | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ---------------- | | `TOWER_ENABLE_OPENAPI` | Enable the OpenAPI documentation endpoint, e.g., [cloud.seqera.io/openapi/index.html](https://cloud.seqera.io/openapi/index.html). | Default: `false` | | `TOWER_RATELIMIT_PERIOD` | Specify the maximum number of HTTP requests that can be made during the `TOWER_RATELIMIT_REFRESH` period. | Default: `20` | | `TOWER_RATELIMIT_REFRESH` | API rate limit refresh period. | Default: `1s` | | `TOWER_RATELIMIT_TIMEOUT` | The waiting period before rejecting requests over the `TOWER_RATELIMIT_PERIOD` limit during the refresh period. | Default: `500ms` | ## OIDC access tokens Configuration values for the audience claim on the OIDC access tokens Seqera Platform issues, and for enforcing it. ::table{file=configtables/oidc_token_env.yml} ## Custom navigation menu Modify your Seqera instance's navigation menu options. ```yaml tower: navbar: menus: - label: "My Community" url: "https://host.com/foo" - label: "My Pipelines" url: "https://other.com/bar" ``` In a [Helm chart](../platform-helm) deployment, set the menu options in the `platform.YAMLConfigFileContent` field of your values file: ```yaml platform: YAMLConfigFileContent: |- tower: navbar: menus: - label: "My Community" url: "https://host.com/foo" - label: "My Pipelines" url: "https://other.com/bar" ``` ## Telemetry Seqera Platform sends usage telemetry to the Seqera License Manager alongside the license check. Telemetry is enabled by default. To opt out, set both `TOWER_TELEMETRY_STANDARD_ENABLED` and `TOWER_TELEMETRY_BASIC_ENABLED` to `false`. ::table{file=configtables/telemetry_env.yml} ## Logging Logging-related configuration values to aid troubleshooting. See [Audit logs](../../monitoring/audit-logs) for more information on application event logging. Use `TOWER_CRON_AUDIT_LOG_CLEAN_UP_ENABLED` to disable automatic audit log deletion, and restart Platform after changing audit log settings. ::table{file=configtables/tower_logging.yml} Set the logging detail level for various Seqera services. Logs for particular services may be requested by support to assist with troubleshooting an issue. Set the logging configuration parameter in your Seqera YAML configuration before attempting to reproduce your issue. The example below sets the detail level for application and database logging: `logger` is a root-level object in the `tower.yml` configuration file, i.e., it is not nested under `tower`. ```yaml logger: levels: org.hibernate.SQL: DEBUG org.hibernate.type: TRACE io.seqera.tower: TRACE ``` ### Distributed tracing Seqera Platform emits OpenTelemetry traces with stable, route-aware span names. You can observe endpoint latency, error rate, and throughput per API route rather than per raw URL. Platform has no tracing switch of its own. Standard OpenTelemetry configuration governs tracing, and the tracing libraries ship in every installation. With no exporter configured, Platform uses an always-off sampler and records no spans. Tracing costs nothing until you turn it on. To send traces to a collector, set the standard exporter variables: ```bash OTEL_TRACES_EXPORTER=otlp OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318 OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf ``` Set `OTEL_SERVICE_NAME`, `OTEL_RESOURCE_ATTRIBUTES`, `OTEL_TRACES_SAMPLER`, and `OTEL_TRACES_SAMPLER_ARG` as appropriate for your deployment. To turn tracing off again, set `OTEL_TRACES_EXPORTER=none`. :::note Platform writes the active trace and span IDs into the logging context. The default [`TOWER_LOG_PATTERN`](#logging) includes `%X{trace_id:--} %X{span_id:--}` so that log lines correlate with traces. If you override the pattern, keep both fields. ::: ### Audit log v2 Configuration values for the v2 audit log subsystem and the audit log cleanup cron job. The v2 audit log adds support for pre/post change state capture and CSV export limits. ::table{file=configtables/audit_log_v2_env.yml} --- ## Pipeline optimization [Pipeline optimization](../../pipeline-optimization/overview) takes the resource usage information from previous workflow runs to optimize subsequent runs. The pipeline optimization service requires a separate database schema to store its internal data, but also requires access to the Seqera schema. The Seqera and optimization service schemas can coexist on the same database instance. ## Docker Compose deployment Docker Compose makes use of a separate container to set up the pipeline optimization service during initialization. Configuration steps differ for new and existing deployments. ### New installation To use the pipeline optimization service in a new Docker Compose installation of Seqera Enterprise, use the following steps: 1. To run the service from a custom URL, declare the URL with the `GROUNDSWELL_SERVER_URL` environment variable in `tower.env`. A non-zero value for this environment variable activates the optimization service automatically, so `TOWER_ENABLE_GROUNDSWELL` does not need to be set when you declare a custom URL. 2. Set the `TOWER_ENABLE_GROUNDSWELL` environment variable in `tower.env` to `true`. This enables the service at the default service URL `http://groundswell:8090`. 3. In your [docker-compose.yml](../_templates/docker/docker-compose.yml) file, uncomment the `groundswell` section at the bottom. - To create a schema for the optimization service on the same local MySQL container, uncomment the `init.sql` script in the `volumes` section. 4. Download the [init.sql](../_templates/docker/init.sql) file. Store this file in the mount path of your `docker-compose.yml` file, else update the `source: ./init.sql` line in your `docker-compose.yml` with the file path. 5. When the pipeline optimization service is active, pipelines that can be optimized display a lightbulb icon in your Launchpad. Any pipeline with at least one successful run can be optimized. ### Existing installation To use the pipeline optimization service in an existing Docker Compose installation of Seqera Enterprise, use the following steps: 1. To run the service from a custom URL, declare the URL with the `GROUNDSWELL_SERVER_URL` environment variable. A non-zero value for this environment variable activates the optimization service automatically, so `TOWER_ENABLE_GROUNDSWELL` does not need to be set when you declare a custom URL. 2. Set the `TOWER_ENABLE_GROUNDSWELL` environment variable to `true`. This enables the service at the default service URL `http://groundswell:8090`. 3. In your [docker-compose.yml](../_templates/docker/docker-compose.yml) file, uncomment the `groundswell` section at the bottom. If you use a `docker-compose.yml` file older than version 23.3, download a newer version of the file to extract the `groundswell` section. 4. Log in to your database server and run the following commands: ```sql CREATE DATABASE IF NOT EXISTS `swell`; CREATE USER 'swell'@'%' IDENTIFIED BY 'swell'; GRANT ALL PRIVILEGES ON *.* TO 'swell'@'%'; FLUSH PRIVILEGES; ``` 5. If you use Amazon RDS or other managed database services, run the following commands in your database instance: ```sql CREATE DATABASE IF NOT EXISTS `swell`; CREATE USER 'swell'@'%' IDENTIFIED BY 'swell'; GRANT ALL PRIVILEGES ON `%`.* TO 'swell'@'%'; FLUSH PRIVILEGES; ``` 6. Download the [groundswell.env](../_templates/docker/groundswell.env) file. Store this file in the mount path of your `docker-compose.yml` file. Update the `TOWER_DB_URL` and `SWELL_DB_URL` values: ```env # Uncomment for container DB instances # TOWER_DB_URL=mysql://db:3306/tower # SWELL_DB_URL=mysql://db:3306/swell # Uncomment for managed DB instances (Example URL shows an Amazon RDS instance URL) # TOWER_DB_URL=mysql://db1.abcdefghijkl.us-east-1.rds.amazonaws.com:3306/tower # SWELL_DB_URL=mysql://db1.abcdefghijkl.us-east-1.rds.amazonaws.com:3306/swell ``` 7. When the pipeline optimization service is active, pipelines that can be optimized display a lightbulb icon in your Launchpad. Any pipeline with at least one successful run can be optimized. ## Kubernetes deployment Kubernetes deployments use an [initContainer](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) that runs during pod initialization to set up the pipeline optimization service. To use the service in new or existing Kubernetes installations of Seqera Enterprise, do the following: 1. Download the [groundswell manifest](../_templates/k8s/groundswell.yml). 1. To run the service from a custom URL, declare the URL with the `GROUNDSWELL_SERVER_URL` environment variable in the `configmap.yml` file that you downloaded for your [Platform installation][platform-k8s]. A non-zero value for this environment variable activates the optimization service automatically, so `TOWER_ENABLE_GROUNDSWELL` does not need to be set when you declare a custom URL. 1. Define a set of credentials for the optimization database. This can be the same database used for Seqera, but in a different schema. 1. Log in to your database server and run the following commands: - If you use Amazon RDS or other managed database services, run the following commands in your database instance: ```sql CREATE DATABASE IF NOT EXISTS `swell`; CREATE USER 'swell'@'%' IDENTIFIED BY 'swell'; GRANT ALL PRIVILEGES ON `%`.* TO 'swell'@'%'; FLUSH PRIVILEGES; ``` - If you do not use a managed database service, run the following commands in your database instance: ```sql CREATE DATABASE IF NOT EXISTS `swell`; CREATE USER 'swell'@'%' IDENTIFIED BY 'swell'; GRANT ALL PRIVILEGES ON *.* TO 'swell'@'%'; FLUSH PRIVILEGES; ``` The `initContainers` process waits until both the Seqera and pipeline optimization service databases are ready before starting the migration in the Seqera database, and finally starting the optimization container. When the pipeline optimization service is active, pipelines that can be optimized display a lightbulb icon in your Launchpad. Any pipeline with at least one successful run can be optimized. [platform-k8s]: ../platform-kubernetes --- ## Reverse proxy :::caution As of February 2024, this configuration guide is not currently recommended for production use, as the instructions are actively under development and will likely change. ::: To expose your Seqera instance behind a reverse proxy, complete the following steps: 1. Add `TOWER_BASE_PATH` to the environment variables of the frontend container: - `TOWER_BASE_PATH: "/myseqera/"` exposes your instance at `https://example.com/myseqera/` (this must match your proxy configuration) 2. In the backend/cron environment variables or in the Seqera config file, edit the following environment variables: - Set `TOWER_SERVER_URL` to the complete URL where you want to expose your instance, e.g., `TOWER_SERVER_URL: "https://example.com/myseqera"` (without the trailing slash) - Disable/unset `TOWER_LANDING_URL` 3. Configure your reverse proxy to redirect all Seqera-related links to your Seqera frontend container: - If your frontend container listens on `http://tower-frontend:8080` and you're using Apache HTTP as your reverse proxy, add the following lines at the end of your configuration file (replace `/myseqera/` with the URL you defined in `TOWER_BASE_PATH`): ``` LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_http_module modules/mod_proxy_http.so LoadModule rewrite_module modules/mod_rewrite.so RewriteEngine on RewriteRule "^/myseqera/(.*)$" http://tower-frontend:8080/$1 [P] ProxyPassReverse "/myseqera/" http://tower-frontend:8080/ RewriteRule "^/api/(.*)$" http://tower-frontend:8080/api/$1 [P] ProxyPassReverse "/api/" http://tower-frontend:8080/api/ RewriteRule "^/auth/(.*)$" http://tower-frontend:8080/auth/$1 [P] ProxyPassReverse "/auth/" http://tower-frontend:8080/auth/ RewriteRule "^/oauth/(.*)$" http://tower-frontend:8080/oauth/$1 [P] ProxyPassReverse "/oauth/" http://tower-frontend:8080/oauth/ RewriteRule "^/openapi/(.*)$" http://tower-frontend:8080/openapi/$1 [P] ProxyPassReverse "/openapi/" http://tower-frontend:8080/openapi/ RewriteRule "^/content/(.*)$" http://tower-frontend:8080/content/$1 [P] ProxyPassReverse "/content/" http://tower-frontend:8080/content/ ``` - A similar configuration should be applied for NGINX or other reverse proxies. Redirect visits to `/api/`, `/oauth/`, `/openapi/`, and `/content/`. After you configure the reverse proxy, the Seqera frontend URL (default `http://tower-frontend:8080`) should return a blank page. This behavior is expected, because Seqera is now configured to work only from behind the reverse proxy. --- ## SSL/TLS HTTP must not be used in production environments. An SSL certificate is required for your Seqera instance to handle HTTPS traffic. Private certificates are supported, but require additional configuration during Seqera Enterprise installation and Nextflow execution. ## AWS deployments: Manage SSL certificates with Amazon Certificate Manager (ACM) Use [Amazon Certificate Manager](https://aws.amazon.com/certificate-manager/) (ACM) to apply SSL certificates to your AWS deployment: - If you have an existing SSL certificate, see [Importing certificates into AWS Certificate Manager](https://docs.aws.amazon.com/acm/latest/userguide/import-certificate.html). - If you don't have an existing SSL certificate, see [Issuing and managing certificates](https://docs.aws.amazon.com/acm/latest/userguide/gs.html). ## Configure Seqera to trust your private certificate If you secure related infrastructure (such as private Git repositories) with certificates issued by a private Certificate Authority, these certificates must be loaded into the Seqera Enterprise containers. You can achieve this in several ways. :::note This procedure loads certificates into the Seqera Enterprise containers. It doesn't reach Studio sessions, which run as separate containers in your compute environment with their own trust store. To let Studio sessions trust an internal certificate authority, see [Configure a private certificate authority for Studios](../studios-private-ca). ::: **Configure private certificate trust** 1. This guide assumes you're using the original containers supplied by Seqera. 2. Replace `TARGET_HOSTNAME`, `TARGET_ALIAS`, and `PRIVATE_CERT.pem` with your unique values. 3. Previous instructions advised using `openssl`. The native `keytool` utility is preferred as it simplifies steps and better accommodates private CA certificates. ### Use Docker volume 1. Retrieve the private certificate on your Seqera container host: ``` keytool -printcert -rfc -sslserver TARGET_HOSTNAME:443 > /PRIVATE_CERT.pem ``` 2. Modify the `backend` and `cron` container configuration blocks in `docker-compose.yml`: ```yaml CONTAINER_NAME: # -- Other keys here like `image` and `networks`-- # Add a new mount for the downloaded certificate volumes: - type: bind source: /PRIVATE_CERT.pem target: /etc/pki/ca-trust/source/anchors/PRIVATE_CERT.pem # Add a new keytool import line PRIOR to 'update-ca-trust' for the certificate command: > sh -c "keytool -import -trustcacerts -storepass changeit -noprompt -alias TARGET_ALIAS -file /etc/pki/ca-trust/source/anchor/TARGET_HOSTNAME.pem && update-ca-trust && /wait-for-it.sh db:3306 -t 60 && /tower.sh" ``` :::caution The two Kubernetes procedures that follow import the certificate directly into the JVM trust store inside the container image. This requires a writable container root filesystem. If you deploy with the Seqera Helm charts, follow **Use Helm charts** instead. ::: ### Use K8s ConfigMap 1. Retrieve the private certificate on a machine with CLI access to your Kubernetes cluster: ```bash keytool -printcert -rfc -sslserver TARGET_HOSTNAME:443 > /PRIVATE_CERT.pem ``` 2. Load the certificate as a `ConfigMap` in the same namespace where your Seqera instance will run: ```bash kubectl create configmap private-cert-pemstore --from-file=/PRIVATE_CERT.pem ``` 3. Modify both the `backend` and `cron` Deployment objects: - Define a new volume based on the certificate `ConfigMap`: ```yaml spec: template: spec: volumes: - name: private-cert-pemstore configMap: name: private-cert-pemstore ``` - Add a volumeMount entry into the container definition: ```yaml spec: template: spec: containers: - name: CONTAINER_NAME volumeMounts: - name: private-cert-pemstore mountPath: /etc/pki/ca-trust/source/anchors/PRIVATE_CERT.pem subPath: PRIVATE_CERT.pem ``` - Modify the container start command to load the certificate prior to running your Seqera instance: ```yaml spec: template: spec: containers: - name: CONTAINER_NAME command: ["/bin/sh"] args: - -c - | keytool -import -trustcacerts -cacerts -storepass changeit -noprompt -alias TARGET_ALIAS -file /PRIVATE_CERT.pem; ./tower.sh ``` ### Download on Pod start 1. Modify both the `backend` and `cron` Deployment objects to retrieve and load the certificate prior to running your Seqera instance: ```yaml spec: template: spec: containers: - name: CONTAINER_NAME command: ["/bin/sh"] args: - -c - | keytool -printcert -rfc -sslserver TARGET_HOST:443 > /PRIVATE_CERT.pem; keytool -import -trustcacerts -cacerts -storepass changeit -noprompt -alias TARGET_ALIAS -file /PRIVATE_CERT.pem; ./tower.sh ``` ### Use Helm charts The Seqera Helm charts set `containerSecurityContext.readOnlyRootFilesystem: true` by default for every component except Studios. Because the root filesystem is read-only, an in-place import into the image's trust store fails when the pod starts: ``` keytool error: java.io.FileNotFoundException: /usr/lib/jvm/JDK_VERSION/lib/security/cacerts (Read-only file system) ``` `keytool` prints `Certificate was added to keystore` before it writes the keystore back to disk. The error appears immediately after this apparent success message. Treat the error, not the success line, as the outcome. Mounted volumes remain writable when `readOnlyRootFilesystem` is enabled. Copy the trust store to a volume, import your certificate into the copy, then point the JVM at the copy. 1. Load the certificate as a `ConfigMap` in the same namespace as your release: ```bash kubectl create configmap private-cert-pemstore --from-file=/PRIVATE_CERT.pem ``` 2. Add the following to your values file. This example configures Wave. The same `command`, `args`, `extraVolumes`, `extraVolumeMounts`, and `extraEnvVars` keys are available for `backend`, `cron`, and the other components. Studios takes them under `studios.proxy` and `studios.server`: ```yaml wave: command: ["/bin/sh"] args: - -c - | cp $JAVA_HOME/lib/security/cacerts /opt/cacerts/cacerts chmod +w /opt/cacerts/cacerts keytool -import -trustcacerts -storepass changeit -noprompt -alias TARGET_ALIAS -file /ca/PRIVATE_CERT.pem -keystore /opt/cacerts/cacerts /launch.sh extraVolumes: - name: cacerts emptyDir: {} - name: private-cert-pemstore configMap: name: private-cert-pemstore extraVolumeMounts: - name: cacerts mountPath: /opt/cacerts - name: private-cert-pemstore mountPath: /ca/PRIVATE_CERT.pem subPath: PRIVATE_CERT.pem extraEnvVars: - name: JAVA_TOOL_OPTIONS value: "-Djavax.net.ssl.trustStore=/opt/cacerts/cacerts" ``` Replace `/launch.sh` with `./tower.sh` for the `backend` and `cron` components. To avoid overriding the container command, copy and import the certificate in an `initContainers` entry that mounts the same `cacerts` volume. 3. Confirm that the JVM received the flag. The pod log prints the following line at startup: ``` Picked up JAVA_TOOL_OPTIONS: -Djavax.net.ssl.trustStore=/opt/cacerts/cacerts ``` If the line is absent, the JVM never received the flag and your trust store has no effect. Check the rendered environment with `kubectl get deploy DEPLOYMENT_NAME -o jsonpath='{.spec.template.spec.containers[0].env}'`. A frequent cause is an `extraEnvVars` entry that is not a list of `name` and `value` maps. Because `- MY_VAR: "some-value"` has no `name` key, the charts render it as `name: ` with no value and the variable you intended is never set. :::note Use `JAVA_TOOL_OPTIONS` rather than a component-specific variable. Because the JVM reads `JAVA_TOOL_OPTIONS` directly, it works for every Seqera Java service. `JAVA_TOOL_OPTIONS` also adds your flag without replacing the JVM options that each container sets by default. The `backend` and `cron` containers read `JAVA_OPTS`, but Wave ignores it and reads `WAVE_JVM_OPTS` instead. Setting `WAVE_JVM_OPTS` replaces the default Wave heap, garbage collector, and Netty options rather than adding to them. ::: :::note Copying the full trust store before importing keeps the public Certificate Authorities trusted and leaves other outbound connections unaffected. If `PRIVATE_CERT.pem` holds a chain of more than one certificate, `keytool -import` adds only the first. Import each certificate under its own alias, or import only the root Certificate Authority that anchors the chain. ::: ## Configure the Nextflow launcher image to trust your private certificate If you secure infrastructure such as private Git repositories or your Seqera Enterprise instance with certificates issued by a private Certificate Authority, these certificates must also be loaded into the Nextflow launcher container. ### Import private certificates via pre-run script 1. This configuration assumes you're using the default `nf-launcher` image supplied by Seqera. 2. Replace `TARGET_HOSTNAME`, `TARGET_ALIAS`, and `PRIVATE_CERT.pem` with your unique values. 3. Previous instructions advised using `openssl`. The native `keytool` utility is preferred as it simplifies steps and better accommodates private CA certificates. Add the following to your compute environment [pre-run script](../../launch/advanced#pre-and-post-run-scripts): ```bash keytool -printcert -rfc -sslserver TARGET_HOSTNAME:443 > /PRIVATE_CERT.pem keytool -import -trustcacerts -cacerts -storepass changeit -noprompt -alias TARGET_ALIAS -file /PRIVATE_CERT.pem cp /PRIVATE_CERT.pem /etc/pki/ca-trust/source/anchors/PRIVATE_CERT.pem update-ca-trust ``` ## Configure Seqera to present a SSL/TLS certificate You can secure your Seqera instance with a TLS certificate in several ways. ### Load balancer (recommended) Place a load balancer, configured to present a certificate and act as a TLS termination point, in front of your Seqera instance. This solution is likely already implemented for cloud-based Kubernetes implementations and can be easily implemented for Docker Compose-based stacks. See [this example](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-application-load-balancer.html). ### Reverse proxy container This solution works well for Docker Compose-based stacks to avoid the additional cost and maintenance of a load balancer. See [this example](https://doc.traefik.io/traefik/v1.7/configuration/acme/). ### Modify `frontend` container Due to complications that can be encountered during upgrades, this approach is not recommended.
Show me anyway This example assumes deployment on an Amazon Linux 2 AMI. 1. Install NGINX and other required packages: ```bash sudo amazon-linux-extras install nginx1.12 sudo wget -r --no-parent -A 'epel-release-*.rpm' https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/ sudo rpm -Uvh dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-*.rpm sudo yum-config-manager --enable epel* sudo yum repolist all sudo amazon-linux-extras install epel -y ``` 2. Generate a [private certificate and key](https://www.digitalocean.com/community/tutorials/openssl-essentials-working-with-ssl-certificates-private-keys-and-csrs). 3. Make a local copy of the `/etc/nginx/templates/tower.conf.template` file from the `frontend` container, or create a ConfigMap to store it if you're using Kubernetes. 4. Replace the `listen` directives in the `server` block with the following: ```nginx listen ${NGINX_LISTEN_PORT} ssl default_server; listen [::]:${NGINX_LISTEN_PORT_IPV6} ssl default_server; ssl_certificate /etc/ssl/testcrt.crt; ssl_certificate_key /etc/ssl/testkey.key; ``` 5. Modify the `frontend` container definition in your `docker-compose.yml` file or Kubernetes manifest: ```yml frontend: image: cr.seqera.io/frontend:${TAG} networks: - frontend environment: NGINX_LISTEN_PORT: 8081 NGINX_LISTEN_PORT_IPV6: 8443 ports: - 8000:8081 - 443:8443 volumes: - $PWD/tower.conf.template:/etc/nginx/templates/tower.conf.template - $PWD/cert/testcrt.crt:/etc/ssl/testcrt.crt - $PWD/cert/testkey.key:/etc/ssl/testkey.key restart: always depends_on: - backend ```
## TLS version support Seqera Enterprise versions 22.3.2 and earlier rely on Java 11 (Amazon Corretto). You may encounter issues when integrating with third-party services that enforce TLS v1.2 (such as Azure Active Directory OIDC). TLS v1.2 can be explicitly enabled by default using JDK environment variables: ```bash JAVA_OPTS="-Dmail.smtp.ssl.protocols=TLSv1.2" ``` --- ## Wave containers Wave is Seqera's container provisioning service that enables on-demand container image management for Nextflow pipelines. Wave can provision containers dynamically during pipeline execution, removing the need to manually build and upload images to a container registry. ## Deployment options Wave can be integrated with Seqera Platform in two ways: - **Seqera Wave service**: Use the hosted Wave service at `https://wave.seqera.io` (default for Seqera Cloud) - **Self-hosted Wave**: Deploy Wave in your own infrastructure for full control over container builds and caching ## Requirements ### Network connectivity | Source | Destination | Purpose | | :----- | :---------- | :------ | | Seqera Platform | Wave server | API communication | | Wave server | Container registry | Container image reads and writes | | Compute environments | Wave server | Container image access during pipeline execution | If you use the Seqera-hosted Wave service and your firewall rules are IP-based, see [firewall configuration](../advanced-topics/firewall-configuration#wave) for the addresses to allow in each direction. Wave reaching your registry uses the `egress` addresses. Your installation reaching Wave uses the `ingress` addresses. ### Container registry credentials Container registry credentials must be configured in the Seqera UI to authenticate with your private or public registries. See [container registry credentials](../../credentials/overview) for provider-specific instructions. ## Configuration ### Connect to Wave service Configure Seqera Platform to use the Seqera-hosted Wave service or your self-hosted Wave deployment: | Variable | Description | | :------- | :---------- | | `TOWER_ENABLE_WAVE` | Set to `true` to enable Wave integration | | `WAVE_SERVER_URL` | Wave server endpoint (default: `https://wave.seqera.io`) | ### Verify connectivity Test connectivity to your Wave server: ```bash curl https://wave.seqera.io/service-info ``` Replace `wave.seqera.io` with your self-hosted Wave endpoint if applicable. ## Features enabled by Wave After Wave is enabled, the following features become available: - **Private container registries**: Access containers from private repositories using credentials stored in Seqera - **Fusion file system**: High-performance cloud-native file system for pipeline execution - **Container augmentation**: Dynamically extend existing containers with additional layers - **Conda-based containers**: Provision containers from Conda or Bioconda packages on demand - **Singularity support**: Build and provision Singularity/Apptainer format containers - **Security scanning**: Automatic vulnerability scanning of built container images Wave features are available on the compute environment creation page after integration is configured. ## Registries behind a private endpoint Wave is a registry proxy. It reads and writes your container images from Seqera's network, not from yours. The Seqera-hosted Wave service cannot reach a registry that is exposed only through a private endpoint inside your own network, such as an Azure Container Registry (ACR) with public network access disabled. The connection fails at name resolution, before Wave evaluates your registry credentials, and valid credentials do not change the outcome. This applies to every Wave mode, including mirror and freeze, which write to your registry. On Azure, the **Allow trusted Microsoft services** setting does not grant Wave access. You have three options: - **Allow Wave's addresses on the registry firewall.** If your security policy allows opening a hole in the firewall, keep the registry's public endpoint enabled and restrict it to the `egress` addresses published at [`meta.seqera.io`](https://meta.seqera.io): see [firewall configuration](../advanced-topics/firewall-configuration#wave). On Azure, registry network rules require the Premium ACR service tier. - **Self-host Wave inside your own network.** The registry stays fully private. See [Self-hosted Wave deployment](#self-hosted-wave-deployment). Wave Lite runs on any conformant Kubernetes distribution, so on Azure it is the only available option: it provides container augmentation, inspection, and private registry authentication. Wave Lite does not provide on-demand builds, freeze, mirroring, or security scanning: those features require the full Wave configuration, which runs on Amazon EKS only. - **Keep the registry fully private and do not use the hosted Wave service.** If Wave hangs without an error when it reaches your registry, see [Wave cannot reach your container registry](../../troubleshooting_and_faqs/troubleshooting#wave-cannot-reach-your-container-registry). ## Limitations - The Seqera-hosted Wave service at `https://wave.seqera.io` cannot pull from container repositories secured by a private Certificate Authority (CA), because you cannot modify its trust store. A self-hosted Wave deployment can pull from these repositories after you add your CA certificate to the trust store in the Wave container. See [Configure Seqera to trust your private certificate](./ssl_tls#configure-seqera-to-trust-your-private-certificate). - The Seqera-hosted Wave service cannot reach a container registry that is exposed only through a private endpoint inside your network. See [Registries behind a private endpoint](#registries-behind-a-private-endpoint). ## Self-hosted Wave deployment For enterprises requiring full control over container builds, caching, and security scanning, Wave can be deployed in your own infrastructure. Self-hosted Wave supports: - **Wave Lite**: Container augmentation, inspection, and private registry authentication. Wave Lite has no AWS dependency and runs on any conformant Kubernetes distribution, or on a single Docker host with Docker Compose. - **Full Wave**: Everything in Wave Lite, plus on-demand container builds, freeze, mirroring, and security scanning. Requires an Amazon EKS installation using EFS for the shared build workspace. See the [Wave documentation](https://docs.seqera.io/wave) for installation and configuration guidance. ## Additional resources - [Wave documentation](https://docs.seqera.io/wave) - [Nextflow Wave integration](https://docs.seqera.io/nextflow/wave) - [Seqera Containers](https://seqera.io/containers/) - Free community container registry --- ## Pipeline optimization: Docker Compose This guide describes how to deploy the pipeline optimization service (referred to as `groundswell` in the configuration file) for Seqera Platform Enterprise using Docker Compose. :::info Prerequisites Other than the basic requirements [already listed in the Pipeline Optimization installation overview](./install-groundswell#prerequisites), you will need: - Docker Engine and Docker Compose ::: ## New installation 1. Set the `TOWER_ENABLE_GROUNDSWELL` environment variable in `tower.env` to `true`. This enables the service at the default URL `http://groundswell:8090`. To use a custom URL, set `GROUNDSWELL_SERVER_URL` instead. 2. In your [docker-compose.yml](./_templates/docker/docker-compose.yml) file, uncomment the `groundswell` section. 3. To create a schema on the local MySQL container, uncomment the `init.sql` script in the `volumes` section. 4. Download the [init.sql](./_templates/docker/init.sql) file and store it in the mount path of your `docker-compose.yml`. 5. Start your Platform instance: ```bash docker compose up -d ``` ## Existing installation 1. Set the `TOWER_ENABLE_GROUNDSWELL` environment variable in `tower.env` to `true`. To use a custom URL, set `GROUNDSWELL_SERVER_URL` instead. 1. In your [docker-compose.yml](./_templates/docker/docker-compose.yml) file, uncomment the `groundswell` section. 1. Download the [groundswell.env](./_templates/docker/groundswell.env) file and update the database URLs: ```env TOWER_DB_URL=mysql://db:3306/tower SWELL_DB_URL=mysql://db:3306/swell ``` 1. Restart your Platform instance: ```bash docker compose up -d ``` ## Verify When pipeline optimization is active, pipelines with at least one successful run display a lightbulb icon in the Launchpad. ## Configuration See [Pipeline optimization](./configuration/pipeline_optimization) for additional configuration options. --- ## Pipeline Optimization: Helm [Helm](https://helm.sh) is an open-source command line tool used for managing Kubernetes applications. Seqera offers a [Helm chart](https://github.com/seqeralabs/helm-charts/tree/pipeline-optimization-0.2.4/platform/charts/pipeline-optimization) to deploy Pipeline Optimization Enterprise on a Kubernetes cluster. :::info Prerequisites Other than the basic requirements [already listed in the Pipeline Optimization installation overview](./install-groundswell#prerequisites), you will need: - A Kubernetes cluster - [Helm v3](https://helm.sh/docs/intro/install) and [kubectl](https://kubernetes.io/docs/tasks/tools/) installed locally ::: ## Installation as part of Seqera Platform Enterprise The Pipeline Optimization Helm chart has been designed as a sub-chart of the main Seqera Platform Enterprise Helm chart, but can optionally be installed independently like the Platform chart. To install Pipeline Optimization as part of your Seqera Platform Enterprise deployment, make sure the `pipeline-optimization.enabled` value in your custom Platform's `values.yaml` file is set to `true`: ```yaml pipeline-optimization: enabled: true ``` At the same time, configure the desired Pipeline Optimization options as described in the [Pipeline Optimization Helm chart documentation](https://github.com/seqeralabs/helm-charts/tree/pipeline-optimization-0.2.4/platform/charts/pipeline-optimization), in particular the Pipeline Optimization and Platform databases. Also refer to the [example](https://github.com/seqeralabs/helm-charts/tree/pipeline-optimization-0.2.4/platform/examples/pipeline-optimization) provided in the Helm charts repository. Then, follow the instructions in the Seqera Platform Enterprise installation guide [using Helm](./platform-helm) to install or upgrade your Platform deployment with Pipeline Optimization. --- ## Pipeline optimization: Kubernetes This guide describes how to deploy the pipeline optimization service (referred to as `groundswell` in the configuration file) for Seqera Platform Enterprise on Kubernetes. :::info Prerequisites Other than the basic requirements [already listed in the Pipeline Optimization installation overview](./install-groundswell#prerequisites), you will need: - A Kubernetes cluster - [kubectl](https://kubernetes.io/docs/tasks/tools/) installed locally ::: ## Procedure 1. Download the [groundswell manifest](./_templates/k8s/groundswell.yml). 1. Set `TOWER_ENABLE_GROUNDSWELL=true` in your `configmap.yml`. To use a custom URL, set `GROUNDSWELL_SERVER_URL` instead. 1. Update the Groundswell ConfigMap (`tower-groundswell-cfg`) with your database credentials. 1. Apply the manifests: ```bash kubectl apply -f configmap.yml kubectl apply -f groundswell.yml ``` 1. Restart the backend: ```bash kubectl rollout restart deployment/backend ``` The initContainers process waits for both databases to be ready before starting the migration and optimization service. ## Verify When pipeline optimization is active, pipelines with at least one successful run display a lightbulb icon in the Launchpad. ## Configuration See [Pipeline optimization](./configuration/pipeline_optimization) for additional configuration options. --- ## Pipeline optimization(Enterprise) Pipeline optimization (Groundswell) uses resource usage data from previous workflow runs to optimize subsequent runs. Deploy after your Platform installation is complete. ## Deployment options | Method | Guide | | :----- | :---- | | Helm | [Pipeline optimization: Helm](./groundswell-helm) | | Kubernetes | [Pipeline optimization: Kubernetes](./groundswell-kubernetes) | | Docker Compose | [Pipeline optimization: Docker Compose](./groundswell-docker-compose) | See each deployment guide for detailed requirements. For an example TLS configuration for the pipeline optimization databases, see the [helm charts repository](https://github.com/seqeralabs/helm-charts/tree/master/charts/platform/examples). ## Prerequisites :::info Before you begin, you need: - A running Seqera Platform Enterprise deployment - A MySQL 8 database separate from the one used by Seqera Platform * For example, to create a new database schema `pipeline_optimization` and user called `pipeline_optimization_admin`, you can run the following SQL commands: ```sql CREATE DATABASE IF NOT EXISTS `pipeline_optimization`; CREATE USER 'pipeline_optimization_admin'@'%' IDENTIFIED BY 'set_a_secure_password_here'; GRANT ALL PRIVILEGES ON `pipeline_optimization`.* TO 'pipeline_optimization_admin'@'%'; FLUSH PRIVILEGES; ``` - Access to the Seqera Enterprise MySQL database (Pipeline Optimization requires direct access to the Seqera database to read workflow execution data) * Read-only access is sufficient * For example, to create a read-only user called `pipeline_optimization_ro` and let it access your Seqera Enterprise database called `seqera_enterprise`, you can run the following SQL commands: ```sql CREATE USER 'pipeline_optimization_ro'@'%' IDENTIFIED BY 'set_a_secure_password_here'; GRANT SELECT ON `seqera_enterprise`.* TO 'pipeline_optimization_ro'@'%'; FLUSH PRIVILEGES; ``` ::: ## Configuration See [Pipeline optimization](./configuration/pipeline_optimization) for additional configuration options. --- ## Platform Seqera Platform Enterprise can be deployed using Docker Compose, Kubernetes, or Helm. ## Deployment options | Method | Use case | | :----- | :------- | | [Helm](./platform-helm) | Kubernetes deployments using Helm charts | | [Kubernetes](./platform-kubernetes) | Production workloads requiring high availability | | [Docker Compose](./platform-docker-compose) | Evaluation, development, small production workloads | See each deployment guide for detailed requirements. ## Prerequisites :::info Before you begin, you need: - A MySQL 8.4 database, the long-term support (LTS) release. MySQL 5.7 and 8.0 have reached upstream end-of-life and are no longer tested or supported. - A Redis 7.2 or 7.4 instance, or a Valkey 7.x instance. Valkey is supported from Seqera Platform 26.1. ::: :::note See each deployment guide for the full version requirements and supported managed-service options for MySQL, Redis, and Valkey. ::: --- ## Co-Scientist(Enterprise) :::caution Co-Scientist requires Seqera Platform Enterprise 25.3.6 or later. This guide covers the Enterprise 26.2 deployment path for the agent backend, MCP server, and Seqera CLI. Sandbox related Co-Scientist features (like file editing, git operations and code execution) are only available on AWS. ::: Deploy the agent backend and Seqera MCP server alongside Platform. Together they provide Co-Scientist assistance for workflows, data, and Platform resources in the [Co-Scientist panel](../co-scientist/platform.md) in Seqera Platform and in the Seqera CLI. Seqera Platform serves the Co-Scientist panel itself. You do not deploy a separate web interface. You can define Kubernetes ingresses in the MCP and agent backend Helm charts, or expose the services in other ways, for example with the `extraDeploy` resource. :::info Enterprise 26.2 removes the `portal-web` subchart and the standalone Co-Scientist web interface. If you deployed `portal-web` with an earlier release, remove the `portal-web` values and its DNS record when you upgrade. Users work in the Co-Scientist panel in Seqera Platform instead. ::: ## Prerequisites Before you begin, make sure you have: - Seqera Platform Enterprise 25.3.6 or later deployed with the [Seqera Platform Helm chart](./platform-helm.md). For Enterprise 26.2, use Platform chart 1.0.x or later. Earlier charts include the removed `portal-web` subchart and are not compatible with 26.2. - Helm v3 and `kubectl` installed locally. - DNS names and TLS certificates for the Platform, agent backend, and MCP server hosts. By default, the Helm charts derive `mcp.` and `ai-api.`. Override `global.mcpDomain` and `global.agentBackendDomain` if you use different hostnames. The agent backend host must share a parent domain with Platform so the Co-Scientist panel can authenticate with the Platform session cookie. - Access to pull the images required by the Helm charts from the configured container registry, or mirrored copies in your internal registry. See [Seqera container images](./advanced-topics/seqera-container-images.md) and [Mirroring container images](./configuration/mirroring.md). - A MySQL 8.4 LTS-compatible database for the agent backend. You can use the same MySQL instance as Platform with a separate database and user, or a separate instance. - A Redis 7.2-compatible or Valkey 7.2-compatible instance for agent backend task coordination. You can use the same instance as Platform with a different Redis database index, or a separate instance. - A stable Fernet token encryption key for the agent backend if you use Kustomize. Helm-only installs can let the chart generate this key, but explicitly setting it avoids accidental regeneration. - Access to a supported Claude inference provider. AWS Bedrock is recommended for Enterprise deployments; direct Anthropic API access is also supported. - If you use AWS Bedrock, an AWS account configured for Co-Scientist. Complete [Bedrock setup](../co-scientist/bedrock-setup.md) before you install: enable model access, submit Anthropic use case details if AWS asks for them, activate the AWS Marketplace agreement for each model, grant the IAM permissions the agent backend pods need, and create an AgentCore runtime if you enable sandboxed sessions. * Note: Certain AWS Accounts have additional account-level eligibility requirements for recent models and may produce errors like `anthropic.claude-opus-4-8 is not available for this account`. These requirements aren't visible in the Service Quotas console, but can be tested by interacting with AWS Bedrock Playground via the console. If that's the case, contact AWS Support to get access to the required models, as explained in [this AWS blog post](https://repost.aws/knowledge-center/bedrock-serverless-models-access-denied). - If you use direct Anthropic API access, an Anthropic API key stored in a Kubernetes Secret. Generate a Fernet token encryption key when you set the key manually: ```bash # using uv Python package manager (installed if not available) uv --version >/dev/null 2>&1 || curl -LsSf https://astral.sh/uv/install.sh | sh uv run --with cryptography python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())" # using Python directly, cryptography dependency module must be installed in environment python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())" ``` Store database passwords, Redis or Valkey passwords, OIDC token values, image pull credentials, and encryption keys in Kubernetes Secrets. Reference those Secrets from Helm values instead of committing plaintext values. ## Inference providers Co-Scientist supports two Claude inference providers. AWS Bedrock is recommended for Enterprise deployments, especially when inference must run in your AWS account or your organization wants to avoid direct egress to Anthropic. Direct Anthropic API access remains supported when your organization has approved that integration. | Inference provider | Description | | --- | --- | | AWS Bedrock | Recommended. Runs Claude inference in your AWS account through Bedrock. See [Bedrock setup](../co-scientist/bedrock-setup.md). | | Anthropic API | Uses Anthropic-hosted Claude models through an Anthropic API key. | Documentation semantic search is configured separately from chat inference. Use Amazon Titan embeddings through Bedrock when you enable improved documentation search. ## Components | Component | Description | | --- | --- | | Agent backend | FastAPI and LangGraph service that orchestrates Co-Scientist sessions, validates Platform tokens, calls the configured inference provider, connects to MCP, and streams Server-Sent Events (SSE) to clients. | | Seqera MCP server | Model Context Protocol server that exposes Platform-aware tools for workflows, datasets, compute environments, Wave, Hub, and nf-core. | | Seqera Platform | Serves the Co-Scientist panel, including page context, conversation history, and [projects](../co-scientist/projects.md). Requires no additional deployment. | | MySQL | Agent backend database for sessions, threads, token usage records, and conversation history. | | Redis or Valkey | Agent backend queue and coordination store. | ## Deployment topology The recommended Enterprise topology uses the `platform` Seqera Helm chart with the `mcp` and `agent-backend` subcharts enabled. The Platform Helm chart automatically wires the MCP OIDC client registration token from the Platform backend secret and sets the Platform configuration the Co-Scientist panel needs. This reduces the number of manual steps. Use separate Helm releases when you cannot convert your existing Platform installation to using the Helm chart or your environment requires separate lifecycle ownership. If you deploy the charts separately, you must manually configure: - `global.platformServiceAddress` and `global.platformServicePort` on each AI chart so they can reach the Platform backend service using the cluster-internal endpoint. - `oidcToken.existingSecretName` and `oidcToken.existingSecretKey` with the same OIDC client registration token configured for the Platform backend. - Matching external DNS and TLS for `global.mcpDomain` and `global.agentBackendDomain`. - `TOWER_AGENT_BACKEND_URL` and `TOWER_AUTH_COOKIE_DOMAIN` on the Platform backend. See [Configure Seqera Platform](#configure-seqera-platform). ## Configure Helm values Enable the two Co-Scientist subcharts in your Platform values file. Because this example uses the Platform parent chart, the same Helm release also deploys or upgrades Platform. Include the required Platform values from your existing installation in addition to these Co-Scientist values. ```yaml global: platformExternalDomain: platform.example.com mcpDomain: mcp.platform.example.com agentBackendDomain: ai-api.platform.example.com mcp: enabled: true agent-backend: enabled: true ``` For a complete example, see the [Co-Scientist Helm example](https://github.com/seqeralabs/helm-charts/tree/master/charts/platform/examples/seqera-co-scientist). ## Configure MCP When MCP runs under the Platform parent chart, leave `mcp.oidcToken` unset unless you need to override the default wiring. The parent chart sets it to the Platform backend secret key `OIDC_CLIENT_REGISTRATION_TOKEN`. The MCP image repository is `enterprise/mcp/server`, pulled from the registry you set in `global.imageRegistry`. The chart sets no default registry. Set `global.imageRegistry` to your own registry after you copy the image from `cr.seqera.io/enterprise/mcp/server`, or to `cr.seqera.io` if your cluster pulls directly. Platform chart 1.0.4 and later default to this repository. Earlier 26.2 charts default to `ai/mcp/server`, which does not host the MCP version they deploy. :::info From MCP 1.4.3, Seqera publishes MCP server images only to `cr.seqera.io/enterprise/mcp/server`. Earlier releases (up to 1.4.2) remain available at `cr.seqera.io/ai/mcp/server`, but Seqera publishes no new releases there. ::: If you mirror images or override the MCP image repository: - Mirror `enterprise/mcp/server` into your registry. New MCP releases are not published to `ai/mcp/server` anymore. - Remove any `mcp.image.repository: ai/mcp/server` override from your values, or change it to `enterprise/mcp/server`. An explicit override takes precedence over the chart default. - If you're using a chart earlier than 1.0.4, upgrade to the latest 1.0.x chart, or set `mcp.image.repository: enterprise/mcp/server`. Seqera does not patch charts. ## Configure the agent backend The agent backend needs MySQL, Redis or Valkey, inference provider access, MCP connectivity, and a stable token encryption key. :::note The agent backend runs as a single replica and is upgraded with the `Recreate` strategy, which stops the running pod before the new one starts. The chart does not let you override `replicas` or `strategy`. Expect a brief Co-Scientist interruption during upgrades. ::: In this example, a Kubernetes secret named `seqera-ai-secrets` stores the sensitive values, such as the database and Redis passwords and the token encryption key. Create this secret before you install the chart, either manually or with an automated secret extraction tool such as External Secrets. This guide does not cover secret extraction tools. Declare which provider serves each capability using `inference.provider`, `embeddings.provider`, and `sandbox.provider`. `inference.provider` is required; `embeddings.provider` and `sandbox.provider` are optional. Leave them empty to disable those features. The `bedrock` block holds credentials and configuration shared across all Bedrock-backed services, with per-service overrides available when needed. :::caution Before you set `inference.provider: bedrock`, complete [Bedrock setup](../co-scientist/bedrock-setup.md). Each model needs model access enabled and an active AWS Marketplace agreement in your account. Otherwise, Co-Scientist's first inference call fails. ::: The following example shows a full Bedrock configuration with embeddings and AgentCore sandbox enabled. For Anthropic inference, see the note after the example. ```yaml agent-backend: enabled: true # -- Database database: host: mysql.example.com name: agent_backend username: agent_backend existingSecretName: seqera-ai-secrets existingSecretKey: AGENT_BACKEND_DB_PASSWORD # Enable when your database requires encrypted connections enableTls: false tlsCaVerify: true # Path to a CA certificate file mounted into the pod sslCa: "" # -- Redis or Valkey redis: host: redis.example.com port: 6379 # Database index. Set a different index to Platform's when both share one instance. database: 0 existingSecretName: seqera-ai-secrets existingSecretKey: AGENT_BACKEND_REDIS_PASSWORD # Enable when your Redis or Valkey instance requires TLS (rediss://) enableTls: false tokenEncryptionKeyExistingSecretName: seqera-ai-secrets # -- Provider routing: declare which provider serves each capability inference: provider: bedrock # required; "bedrock" or "anthropic" embeddings: provider: bedrock # optional; omit to disable documentation search sandbox: provider: bedrock # optional; omit to disable AgentCore sandbox sessions # -- Bedrock configuration. See Bedrock setup for the IAM policies these roles require. bedrock: # Default credentials applied to all Bedrock-backed services unless overridden per-service. # Use this when inference, embeddings, and sandbox all share the same role and region. default: assumeRoleArn: arn:aws:iam:::role/ region: inference: # Anthropic inference profile ARN on Bedrock. anthropicModel: arn:aws:bedrock:::inference-profile/ embeddings: model: amazon.titan-embed-text-v2:0 sandbox: # AgentCore runtime ARN — required when sandbox.provider is "bedrock". runtimeArn: arn:aws:bedrock-agentcore:::runtime/ ``` Use `bedrock.default.assumeRoleArn` when the pod must assume a role to access Bedrock services. Leave it empty when the pod already has direct AWS credentials for the target account. Per-service overrides (`bedrock.inference.assumeRoleArn`, `bedrock.embeddings.assumeRoleArn`, `bedrock.sandbox.assumeRoleArn`) are available when different roles are required per capability. To use direct Anthropic API access instead of Bedrock for inference, replace the `inference` and `bedrock.inference` blocks in the previous example with the following, and add the `anthropic` block. From `platform` chart 1.0.5, which ships `agent-backend` subchart 1.6.0, you can configure the Anthropic inference model directly in the chart. You can still enable Bedrock embeddings alongside Anthropic inference: ```yaml inference: provider: anthropic anthropic: existingSecretName: seqera-ai-secrets # Platform chart 1.0.5 (shipping agent-backend subchart 1.6.0) provides the option to configure the Anthropic inference model directly. # Alternatively, you can configure the Anthropic inference model by setting the `ANTHROPIC_MODEL` environment variable on the agent backend. inference: model: claude-opus-4-8 embeddings: provider: bedrock bedrock: default: assumeRoleArn: arn:aws:iam:::role/ region: embeddings: model: amazon.titan-embed-text-v2:0 ``` Use direct Anthropic API access only when your organization has approved Anthropic-hosted Claude models. ### Service account The chart creates a Kubernetes ServiceAccount for the agent backend by default. To use an existing ServiceAccount, for example one bound to an IAM role through EKS Pod Identity or IRSA, set: ```yaml agent-backend: serviceAccount: create: false name: ``` ### Session retention and limits The agent backend deletes CLI sessions after 14 days and Co-Scientist panel conversations after 180 days, and caps sessions at 100 per user and 500 per workspace. To change these defaults, set environment variables on the agent backend with `extraEnvVars`: ```yaml agent-backend: extraEnvVars: - name: CLI_SESSION_RETENTION_DAYS value: "14" - name: WEB_SESSION_RETENTION_DAYS value: "180" - name: MAX_SESSIONS_PER_USER value: "100" - name: MAX_SESSIONS_PER_WORKSPACE value: "500" ``` ### GitHub access Co-Scientist connects to GitHub through a GitHub App that you register for your installation. When GitHub access is configured, each user connects their own GitHub account from the Co-Scientist panel the first time Co-Scientist needs access to a private repository. Co-Scientist can then clone repositories, create branches, push commits, and open pull requests on the user's behalf. GitHub access is off by default. Until you configure it, Co-Scientist cannot connect to users' GitHub accounts. This GitHub App is separate from the [GitHub App credentials](../git/overview.md#github-app) that Seqera Platform uses for pipelines and that Co-Scientist [agents](../co-scientist/agents.md#access-to-private-git-repositories) use. Those credentials do not give the Co-Scientist panel access to GitHub. GitHub access requires: - A sandbox provider, set with `sandbox.provider`. Co-Scientist runs Git operations in the sandbox. - Outbound HTTPS access from the agent backend to `github.com` and `api.github.com`. Co-Scientist supports GitHub.com only, not GitHub Enterprise Server. To configure GitHub access: 1. On GitHub, [register a GitHub App](https://docs.github.com/en/apps/creating-github-apps/registering-a-github-app/registering-a-github-app) for your organization with these settings: - **Callback URL**: `https:///v1/github/oauth/callback`, for example `https://ai-api.platform.example.com/v1/github/oauth/callback`. - **Webhook**: Deselect **Active**. Co-Scientist does not use webhook events. - **Permissions**: Under repository permissions, set **Contents** and **Pull requests** to **Read and write**, and **Metadata** to **Read-only**. Under account permissions, set **Email addresses** to **Read-only**. - **Where can this GitHub App be installed?**: Select **Any account** if users need repositories in more than one GitHub organization. 1. On the app's settings page, copy the client ID and generate a client secret. Note the app slug, which is the last part of the app's public URL: `https://github.com/apps/`. 1. Add the client secret to your Kubernetes secret, for example `seqera-ai-secrets`, under the key `GITHUB_APP_CLIENT_SECRET`. 1. Set the GitHub App variables on the agent backend with `extraEnvVars`. If you already set `extraEnvVars`, for example for [session retention](#session-retention-and-limits), add these entries to the same list: ```yaml agent-backend: extraEnvVars: - name: GITHUB_APP_GITHUB_CONNECT_TOOLS_ENABLED value: "true" - name: GITHUB_APP_CLIENT_ID value: "" - name: GITHUB_APP_SLUG value: "" - name: GITHUB_APP_CLIENT_SECRET valueFrom: secretKeyRef: name: seqera-ai-secrets key: GITHUB_APP_CLIENT_SECRET ``` 1. Apply the change with `helm upgrade`. See [Install or upgrade](#install-or-upgrade). 1. Install the app on each GitHub organization whose repositories users need, at `https://github.com/apps//installations/new`. An organization admin must install the app. If a user asks Co-Scientist to work with a repository in an organization where the app is not installed, Co-Scientist tells the user and returns this link. ## Configure Seqera Platform When the `agent-backend` subchart is enabled, the Platform Helm chart sets the Platform configuration the Co-Scientist panel needs: - `TOWER_AGENT_BACKEND_URL`, set to `https://`. - `TOWER_AUTH_COOKIE_DOMAIN`, set to `.`, so the panel can authenticate to the agent backend with the Platform session cookie. If you deploy the Co-Scientist charts separately from Platform, set both values on the Platform backend yourself. Once `TOWER_AGENT_BACKEND_URL` is set, Seqera Platform enables the Co-Scientist panel for every organization in the installation. Before you upgrade, review whether this is appropriate. The panel sends page context, including screenshots of the page the user is viewing, to the agent backend and your inference provider. To limit the panel to specific organizations, set `TOWER_AI_CHAT_ALLOWED_ORGANIZATIONS` to a comma-separated list of organization IDs. Users also need the `chat:execute` permission. The predefined Owner, Admin, Maintain, Launch, Connect, and Project workspace roles include it. The View role and custom roles do not include it unless you add it. See [Custom roles](../orgs-and-teams/custom-roles.md#ai). [Projects](../co-scientist/projects.md) are enabled by default in every organization workspace and do not depend on the agent backend. To restrict them, set `TOWER_SCIENTIST_VIEW_ALLOWED_WORKSPACES` to a comma-separated list of workspace IDs. For values and full descriptions, see [Co-Scientist configuration](./configuration/overview.mdx#co-scientist). ## Install or upgrade Run Helm with your Platform values and Co-Scientist overrides: ```bash helm upgrade --install seqera oci://public.cr.seqera.io/charts/platform \ --namespace seqera \ --values values.yaml ``` After installation, verify the pods are ready: ```bash kubectl get pods -n seqera -l app.kubernetes.io/component=mcp kubectl get pods -n seqera -l app.kubernetes.io/component=agent-backend ``` ## Verify the installation Check the public endpoints: ```bash curl -i https://ai-api.platform.example.com/health curl -i https://mcp.platform.example.com/health curl -i https://mcp.platform.example.com/service-info ``` The agent backend `/health` endpoint returns `200 OK` when the service starts and required dependencies are reachable. The agent backend also exposes `/live` and `/ready` for Kubernetes probes. The MCP server exposes `/health` for reachability and `/service-info` for server and protocol information. Sign in to Seqera Platform, open an organization workspace, and select **Co-Scientist** in the navigation bar. If the panel opens and answers a question, Platform, the agent backend, and the inference provider are connected. If you configured sandboxing, ask a question that triggers a sandbox execution, for example `What's the accurate square root of 98723516236?`. The model should write a small Python script that runs in the sandbox. ### Run deployment diagnostics To test every subsystem at once, run the `/doctor` skill from the Co-Scientist panel or from a CLI session. It checks the agent's tools, MCP and Platform connectivity, and local (CLI) or sandbox (panel) code execution, and reports PASS, FAIL, or SKIP per subsystem with remediation steps. For a scriptable check, enable the agent backend service-info endpoint. It is disabled by default because it exposes internal dependency topology: ```yaml agent-backend: extraEnvVars: - name: SERVICE_INFO_ENABLED value: "true" ``` Then query it: ```bash # Connectivity for the database, Redis, Platform, MCP, inference, and sandbox configuration curl -s https://ai-api.platform.example.com/v1/service-info # Live inference and sandbox checks (requires a Platform access token) curl -s -H "Authorization: Bearer " \ "https://ai-api.platform.example.com/v1/service-info?deep=1" ``` ## Connect the Seqera CLI to Co-Scientist Install the CLI from the official [`seqera` npm package](https://www.npmjs.com/package/seqera): ```bash npm install -g seqera ``` Point the CLI at your Enterprise deployment: ```bash export SEQERA_AUTH_DOMAIN=https://platform.example.com/api export SEQERA_AI_BACKEND_URL=https://ai-api.platform.example.com seqera ai ``` Alternatively, distribute a `~/.config/seqera-ai/config.json` file to your users with the same values: ```json { "authDomain": "https://platform.example.com/api", "backendUrl": "https://ai-api.platform.example.com" } ``` Set `SEQERA_AUTH_CLI_CLIENT_ID` only if your deployment uses a CLI OAuth client ID other than the default `seqera_ai_cli`. For automated environments, use a Platform access token instead of browser login. Current CLI builds still require `SEQERA_AUTH_DOMAIN` so the CLI can target the correct Enterprise Platform authority. ```bash export SEQERA_AUTH_DOMAIN=https://platform.example.com/api export TOWER_ACCESS_TOKEN= export SEQERA_AI_BACKEND_URL=https://ai-api.platform.example.com seqera ai ``` The CLI supports `SEQERA_ACCESS_TOKEN` and `TOWER_ACCESS_TOKEN` for token-based authentication. Run `seqera info` to confirm which Platform and agent backend the CLI uses. See [Installation](../co-scientist/installation.mdx) for the user-facing steps. ## Usage and cost Usage and inference costs are managed by your organization through the configured inference provider, such as AWS Bedrock or Anthropic API. ## Security considerations - Use HTTPS for every exposed hostname. - Store all sensitive values in Kubernetes Secrets. - Keep the agent backend Fernet token encryption key stable across upgrades. Changing it prevents the backend from decrypting existing encrypted values. - For user-scoped operations, MCP uses the signed-in user's Platform token to call Platform APIs. Do not configure a shared administrator token for these calls. - Use a separate MySQL database and user for the agent backend, even if they are hosted on the same MySQL instance as Platform. - Enable Redis or Valkey TLS (`redis.enableTls`) and MySQL TLS (`database.enableTls`, with `database.sslCa` for a private CA) when your managed services require encrypted connections. - Leave the service-info endpoint disabled unless you need it, and restrict access to it at your ingress. ## Learn more - [Co-Scientist](../co-scientist/index.md): Co-Scientist documentation. - [Bedrock setup](../co-scientist/bedrock-setup.md): Bedrock model access, Marketplace agreements, IAM permissions, and the AgentCore runtime. - [Co-Scientist Helm example](https://github.com/seqeralabs/helm-charts/tree/master/charts/platform/examples/seqera-co-scientist): Example Platform values for the Co-Scientist subcharts. - [Agent backend chart](https://github.com/seqeralabs/helm-charts/tree/master/charts/platform/charts/agent-backend): Full agent backend values reference. - [MCP chart](https://github.com/seqeralabs/helm-charts/tree/master/charts/platform/charts/mcp): Full MCP values reference. --- ## Studios Studios provides interactive analysis environments within Seqera Platform. Deploy Studios after your Platform installation is complete. ## Deployment options | Method | Guide | | :----- | :---- | | Helm | [Studios: Helm](./studios-helm) | | Kubernetes | [Studios: Kubernetes](./studios-kubernetes) | | Docker Compose | [Studios: Docker Compose](./studios-docker-compose) | See each deployment guide for detailed requirements. ## Prerequisites :::info Before you begin, you need: - A running Seqera Platform Enterprise deployment at hostname `example.com` or `platform.example.com` - A Redis 7 instance separate from the one used by Seqera Platform - TLS certificates for the Studios subdomains `*.connect.example.com` * A single certificate covering both Platform and Studios subdomains can be used; if preferring to use separate certificates, you may need to configure a separate ingress when working with plain Kubernetes manifests (the Studios Helm chart already uses separate ingresses) * The Studios subdomain must share the same "root domain" with the Platform installation, for example with Platform installed at `example.com` or `platform.example.com`, Studios can be installed at: - `connect.example.com` or using another name such as `studios.example.com` - `connect.platform.example.com` - `connect.another.subdomain.example.com` - A wildcard DNS record covering the Studios subdomains, e.g., `*.connect.example.com` - Data Explorer enabled in your Seqera Platform instance (automatic with Helm deployments) ::: ## Connect environment variables These are the environment variables used to configure the components of Connect. | Environment variable | Default | Required | Used by | Description | |------------------------------------------|-----------------------------|------------|--------------|---------------------------------------------------------------------------------------------------| | `CONNECT_REDIS_ADDRESS` | `redis:6379` | yes | server,proxy | The address of the Redis server. Default applies to the server; the proxy requires it explicitly. | | `CONNECT_REDIS_USER` | | no | server,proxy | The username to authenticate with Redis. | | `CONNECT_REDIS_PASSWORD` | | no | server,proxy | The password to authenticate with Redis. | | `CONNECT_REDIS_DB` | `0` | no | server,proxy | The Redis database to use. | | `CONNECT_REDIS_PREFIX` | `connect:session` | no | server,proxy | A prefix to use for tunnel keys in Redis. | | `CONNECT_REDIS_TLS_ENABLE` | `false` | no | server,proxy | Enable TLS connection. | | `CONNECT_REDIS_TLS_SKIP_VERIFY` | `false` | no | server,proxy | Sets the insecure skip verify TLS option. | | `CONNECT_REDIS_TLS_KEY_FILE` | | no | server,proxy | The path to a certificate key file for TLS connection. | | `CONNECT_REDIS_TLS_CERT_FILE` | | no | server,proxy | The path to a certificate file for TLS connection. | | `CONNECT_LISTENER_PORT` | `7777` | no | server | The port where the server listens for connections. | | `CONNECT_TUNNEL_PORT` | `7070` | no | server | The port to open a new tunnel. | | `CONNECT_MANAGEMENT_PORT` | | no | server,proxy | The port where the server listens for metrics, readiness, and shutdown. | | `CONNECT_MANAGEMENT_AUTH_KEY` | | no | server | Auth key protecting the management service endpoints. | | `CONNECT_HOST_DOMAIN` | | no | server | The host domain suffix for the server. | | `CONNECT_HTTP_PORT` | `80` | no | proxy | The port where the proxy listens for incoming connections. | | `CONNECT_PROXY_URL` | | yes | proxy | The base domain name of Connect. | | `CONNECT_TUNNEL_URL` | | yes | proxy | The address of the connect server. Format: `:`. | | `PLATFORM_URL` | | yes | proxy | The base URL of Seqera Platform. | | `CONNECT_STORAGE_ROOT` | `/data` | no | proxy | The root directory to store the proxy data. | | `CONNECT_LOG_LEVEL` | `INFO` | no | server,proxy | Log level for the server and proxy. | | `CONNECT_CLIENT_NAME` | `tower-connect-proxy-client` | no | proxy | OIDC client name used by the proxy's Studio provider. | | `CONNECT_GRANT_TYPE` | `authorization_code` | no | proxy | OAuth grant type used by the proxy's Studio provider. | | `CONNECT_OIDC_CLIENT_REGISTRATION_TOKEN` | | no | proxy | OIDC initial access token used by the proxy. | | `LOCAL_CACHE_TTL` | `2m` | no | proxy | TTL for the proxy's local session cache before syncing with redis. | | `CONNECT_SSH_ENABLED` | `false` | no | proxy | Enable the SSH proxy server. | | `CONNECT_SSH_ADDR` | `:2222` | no | proxy | The address the SSH proxy server listens on. | | `CONNECT_SSH_KEY_PATH` | | no | proxy | Path to SSH host key file. Takes precedence over `CONNECT_SSH_KEY_VALUE_BASE64` when set. | | `CONNECT_SSH_KEY_VALUE_BASE64` | | no | proxy | Base64-encoded PEM SSH host key. Used as fallback when `CONNECT_SSH_KEY_PATH` is not set. | | `CONNECT_SSH_MAX_CONNECTIONS` | `2000` | no | proxy | Max number of concurrent ssh connections that the server will handle before start rejecting them. | | `CONNECT_SSH_MAX_CONN_CHANNELS` | `30` | no | proxy | Max number of concurrent channels that a client can open per connection. | | `CONNECT_SSH_HANDSHAKE_TIMEOUT` | `1m` | no | proxy | SSH handshake timeout. | | `CONNECT_TRUSTED_PROXY_CIDRS` | `127.0.0.1/32` | no | proxy | Space-separated CIDRs of trusted upstream proxies, used to resolve the real client IP from `X-Forwarded-For` for per-IP telemetry. Default is a no-op loopback range. See [Studios data transfer quotas](./studios-transfer-quotas). | | `CONNECT_POLICY_B64` | | no | proxy | Base64-encoded JSON traffic policy. Empty or unset disables telemetry and quota enforcement. Mutually exclusive with `CONNECT_POLICY_FILE`. See [Studios data transfer quotas](./studios-transfer-quotas). | | `CONNECT_POLICY_FILE` | | no | proxy | Path to a JSON traffic policy file. Mutually exclusive with `CONNECT_POLICY_B64`. | | `CONNECT_TELEMETRY_FLUSH_INTERVAL` | `30s` | no | proxy | How often the proxy writes in-memory byte counters to Redis. | | `CONNECT_TELEMETRY_TTL` | `168h` | no | proxy | TTL for cumulative per-bucket telemetry keys in Redis (7 days). | | `CONNECT_TELEMETRY_STREAM_EMIT_INTERVAL` | `1s` | no | proxy | How often long-lived streams (WebSocket and SSH) report transferred bytes. | | `TOWER_CONNECT_CA_CERT_BASE64` | | no | client | Base64-encoded internal CA certificate that the client installs in the session trust store. Platform sets this automatically from `TOWER_SSL_CUSTOM_CA_CERT_FILE`. Do not set it by hand. Requires client 0.13.0 or later. See [Configure a private certificate authority for Studios](./studios-private-ca). | | `TOWER_CONNECT_CA_KEEP_DEFAULT` | `true` | no | client | Whether the session keeps the container image's public CAs alongside your internal CA. Set to `false` to restrict trust to your CA alone. See [Configure a private certificate authority for Studios](./studios-private-ca). | :::note You supply the internal CA to Platform, not to Connect. Set `TOWER_SSL_CUSTOM_CA_CERT_FILE` on the Platform deployment, and Platform provisions the certificate into every Studio session. See [Configure a private certificate authority for Studios](./studios-private-ca). ::: ## DNS configuration Each Studio is reachable at a unique URL that includes a randomly generated subdomain name. For example: `https://abcd.connect.example.com/`, where `connect.example.com` is the Studios service domain. Provide a wildcard TLS certificate to allow for uniquely generated subdomains. A wildcard certificate common name includes `*.` in the domain name, such as `*.connect.example.com`, thereby securing any subdomain name at this level. Studios uses the following set of domains and subdomains: - The Platform domain that you set for `TOWER_SERVER_URL`, such as `example.com`. - A wildcard subdomain that you must configure specifically for Studios. This wildcard subdomain is the parent for each unique session URL, such as `abcd.connect.example.com`. - The connection proxy, defined by `CONNECT_PROXY_URL`. This URL is a first-level subdomain of your `TOWER_SERVER_URL`. For example, `https://connect.example.com`. ## Studios workspace availability You can configure which organizational workspaces have access to Studios by setting the `TOWER_DATA_STUDIO_ALLOWED_WORKSPACES` environment variable on the backend containers. By default, all workspaces have access to Studios. To restrict access to specific workspaces, set `TOWER_DATA_STUDIO_ALLOWED_WORKSPACES` to a comma-separated list of workspace names. For example, `TOWER_DATA_STUDIO_ALLOWED_WORKSPACES="12345,67890"` allows only the workspaces named `12345` and `67890` to access Studios. To disable access to Studios for all workspaces, set `TOWER_DATA_STUDIO_ALLOWED_WORKSPACES=""` (an empty string). ## Available Studios environment images Each of the provided environments includes a particular version of the underlying software package and the version of Seqera Connect, an integrated web- and file-server. To quickly identify which version of the software an image includes, the version string for each container is in the form of `-`. For example, if the version string for the R-IDE is `2025.04.1-0.12`, version `2025.04.01` is the R-IDE version and `0.12` is the Connect version of this Seqera-built container image. Learn more about Studios [environment versioning](../studios/container-images). - To see the list of all JupyterLab image templates available, including security scan results or to inspect the container specification (including container specifications, configuration, and manifest), see [public.cr.seqera.io/repo/platform/data-studio-jupyter][ds-jupyter]. - To see the list of all R-IDE image templates available, including security scan results or to inspect the container specification (including container specifications, configuration, and manifest), see [https://public.cr.seqera.io/repo/platform/data-studio-ride][ds-ride]. - To see the list of all Visual Studio Code image templates available, including security scan results or to inspect the container specification (including container specifications, configuration, and manifest), see [public.cr.seqera.io/platform/data-studio-vscode][ds-vscode]. - To see the list of all Xpra image templates available, including security scan results or to inspect the container specification (including container specifications, configuration, and manifest), see [public.cr.seqera.io/repo/platform/data-studio-xpra][ds-xpra]. ## Path-based routing configuration If your Enterprise deployment requires non-wildcard SSL certificates, enable path-based routing for Studios. This changes the dynamic subdomain used for each Studios session to a fixed subdomain with path-based routing. - When `TOWER_DATA_STUDIO_ENABLE_PATH_ROUTING` is omitted, empty, or `false`, the Studios session URLs use unique subdomains: - https://a1234abc.connect.cloud.seqera.io/ - https://a5678abcd.connect.cloud.seqera.io/ - When `TOWER_DATA_STUDIO_ENABLE_PATH_ROUTING=true`, the Studios session URLs use path-based routing: - https://connect.connect.cloud.seqera.io/_studio/a1234abc - https://connect.connect.cloud.seqera.io/_studio/a5678abcd Path-based routing is only available from Seqera Platform version 25.2 and the latest Connect server and clients. It is supported for Visual Studio Code, JupyterLab, and R-IDE container template images. It is not supported for the Xpra container template image. {/* links */} [ds-jupyter]: https://public.cr.seqera.io/repo/platform/data-studio-jupyter [ds-ride]: https://public.cr.seqera.io/repo/platform/data-studio-ride [ds-vscode]: https://public.cr.seqera.io/repo/platform/data-studio-vscode [ds-xpra]: https://public.cr.seqera.io/repo/platform/data-studio-xpra --- ## Enterprise installation :::tip Seqera Enterprise requires a license. If you have not already purchased a license, [contact us](https://seqera.io/contact-us/) for more information. ::: Seqera Platform Enterprise is a web application with a microservice-oriented architecture that is designed to maximize portability, scalability, and security. It's composed of several modules that are configured and deployed according to your organizational requirements. Seqera provides these modules as Docker container images that are securely hosted on a private container registry. ## Architecture ![Platform architecture diagram](./_images/seqera_reference_architecture2.png) ### Platform backend The Seqera backend is a JVM-based web application based on the [Micronaut](https://micronaut.io/) framework, which provides a modern and secure backbone for the application. The backend implements the main application logic, which is exposed via a REST API and defined with an OpenAPI schema. The backend uses JPA, Hibernate, and JDBC API industry standards to interact with the underlying relational database. The backend can be run standalone or as multiple replicas for scalability when deployed in high-availability mode. It should run on port `8080`. ### Platform cron Cron is an auxiliary backend service that executes regularly-occurring activities, such as sending email notifications and cleaning up stale data. The cron service also performs database migrations at startup. ### Platform frontend The Seqera frontend is an NGINX web server that serves the [Angular](https://angular.io/) application and reverse-proxies HTTP traffic to the backend. The frontend should run on port `80` within the container and should be the only service that accepts incoming HTTP traffic. The frontend can also be exposed via HTTPS or a load balancer. ### Redis or Valkey cache Seqera Enterprise requires a Redis-compatible cache. From Seqera Enterprise 26.1, it supports only Redis 7.2 and 7.4 and Valkey 7.x. See [Cache layer changes](./upgrade#cache-layer-changes-redis-eol-and-valkey-support) for migration guidance. ### SQL database Seqera requires a SQL database to persist user activities and state. MySQL 8.4, the long-term support (LTS) release, is the recommended version. [Contact Seqera support](https://support.seqera.io) if you need to use a different JDBC-compliant SQL database. :::note - MySQL 8.4 is the tested database version from Seqera Enterprise 26.1. MySQL 5.7 and 8.0 have reached upstream end-of-life and are no longer tested or supported. Migrate to MySQL 8.4 before you upgrade to 26.1. See [Database changes](./upgrade#database-changes) for the full support matrix, including AWS Aurora MySQL and MariaDB. ::: ### SMTP service Seqera requires an SMTP relay to send email messages and user notifications. ### Authentication service (optional) Seqera supports enterprise authentication mechanisms such as OAuth and OpenID. Third-party identity providers and custom single sign-on flows can be developed according to specific customer requirements. ## Deployment options Deploy Seqera Enterprise to a single node with [Docker Compose](./platform-docker-compose), or to a [Kubernetes](./platform-kubernetes) cluster with manifests or the [Helm chart](./platform-helm). This documentation includes instructions for all options across multiple platforms, including Amazon AWS, Microsoft Azure, Google Cloud, and on-prem infrastructure. ### Single-node The minimal Seqera Enterprise deployment requires only the frontend, backend, and database services. These services can be deployed as Docker containers or as native services. ### Kubernetes Kubernetes is the recommended deployment target for production workloads that require high availability and scalability. Amazon Elastic Kubernetes Service (EKS) is the preferred Kubernetes platform for Seqera Enterprise. Deploy Seqera Enterprise to Kubernetes with the manifests or the Seqera Helm chart. ![Reference architecture for Seqera Platform Enterprise on AWS EKS](./_images/seqera_reference_architecture_aws.png) _Reference architecture diagram of Seqera Platform Enterprise on AWS using Elastic Kubernetes Service (EKS)_ ## Application container images Seqera Enterprise is distributed as a collection of Docker containers available through the Seqera container registry [`cr.seqera.io`](https://cr.seqera.io). Contact [support](https://support.seqera.io) to get your container access credentials. ## Support For further information, [contact Seqera support](https://support.seqera.io). --- ## Platform: Docker Compose Docker Compose deployments are suitable for evaluation, development, and small production workloads. :::info Prerequisites Other than the basic requirements [already listed in the Platform installation overview](./install-platform#prerequisites), you will need: - Docker Engine and Docker Compose ::: ## Container images Seqera Enterprise container images are hosted on a private registry (`cr.seqera.io`). Access is provided as part of your purchase. Contact [support](https://support.seqera.io) if you require access. We recommend mirroring these images to your own private container registry for production use. See [Mirroring container images](./configuration/mirroring) for details. ## Database configuration Create a MySQL database and user for Seqera: ```sql CREATE DATABASE tower; CREATE USER 'tower'@'%' IDENTIFIED BY 'your_secure_password'; GRANT ALL PRIVILEGES ON tower.* TO 'tower'@'%'; ``` See [Database configuration](./configuration/overview#seqera-and-redis-databases) for details. ## Redis or Valkey configuration Seqera Platform requires a Redis-compatible cache store for transient data, primarily Nextflow job metrics reporting. Both Redis and Valkey are supported. :::info The bundled `redis` container in `docker-compose.yml` is intended for evaluation and small workloads. For production, use a managed service or an [official Redis installation source](https://redis.io/docs/latest/operate/oss_and_stack/install/). ::: ### Supported versions | Cache / version | Status | | --------------- | ---------------------------- | | Redis 6.x | Not supported (from 26.1) | | Redis 7.2 | Supported | | Redis 7.4 | Supported | | Valkey 7.x | Supported (from 26.1) | :::note Managed cache services can offer newer engine versions than Seqera Platform is tested against. For example, Amazon ElastiCache offers Valkey up to 9.x. Select a version from the table. Newer major versions are not tested or supported. ::: ### Connection URL Configure the connection URL in your Seqera environment using the scheme that matches your cache backend: | Backend | Scheme | Example | | --------------- | ------------ | ---------------------------------------- | | Redis | `redis://` | `TOWER_REDIS_URL=redis://:6379` | | Redis with TLS | `rediss://` | `TOWER_REDIS_URL=rediss://:6380` | Seqera Platform 26.1 and later supports the Valkey 7 dial schema. No further configuration is required. Redis password and ACL configuration carry over unchanged when you migrate to Valkey. ### Managed service options Use a managed cache service for production: - [Amazon ElastiCache](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/WhatIs.html) (`cache.m4.large` or larger) - [Azure Managed Redis](https://learn.microsoft.com/azure/redis/overview) (production-capable tier appropriate for your workload) - [Google Memorystore](https://cloud.google.com/memorystore/docs/redis) (M2 tier or larger) :::caution Microsoft is retiring Azure Cache for Redis. As of April 1, 2026, new customers cannot create instances, and from October 1, 2026, no new instances can be created. For new Azure deployments, use [Azure Managed Redis](https://learn.microsoft.com/azure/redis/overview). ::: For migration guidance from Redis to Valkey on an existing installation, see [Cache layer changes](./upgrade#cache-layer-changes-redis-eol-and-valkey-support). ## Deploy Seqera Enterprise 1. Download and configure [tower.env](_templates/docker/tower.env). See [Configuration](./configuration/overview.mdx#basic-configuration) for detailed instructions. 2. Download and configure [tower.yml](_templates/docker/tower.yml). See [Configuration](./configuration/overview.mdx#basic-configuration) for detailed instructions. 3. Download and configure the [docker-compose.yml](_templates/docker/docker-compose.yml) file: - The `db` and `redis` containers should be used only for local testing. If you have configured these services elsewhere, you can remove these containers. - To configure the Seqera pipeline optimization service (`groundswell`), see [Pipeline optimization](./configuration/pipeline_optimization). - To deploy with Studios, see [Studios deployment](./install-studios). 4. Deploy the application and wait for it to initialize (this process takes a few minutes): ```bash docker compose up ``` 5. [Test](./testing) the application by running an nf-core pipeline with a test profile. 6. After you've confirmed that Seqera Enterprise is correctly configured and you can launch workflows, run `docker compose up -d` to deploy the application as a background process. You can then disconnect from the VM instance. :::note For more information on configuration, see [Configuration options](./configuration/overview.mdx). ::: #### Seqera frontend unprivileged From Seqera Platform Enterprise 26.2, Seqera publishes one frontend image, `cr.seqera.io/enterprise/platform/frontend:`. The image is unprivileged. It listens on an unprivileged port and doesn't run as the root user. Seqera no longer publishes the `-root` and `-unprivileged` tag variants. A `docker-compose.yml` that references one fails to pull. The `docker-compose.yml` template earlier on this page is already configured for this image. To listen on a port other than the default `8000`, set `NGINX_LISTEN_PORT` and map the host port to it: ```yaml frontend: image: cr.seqera.io/enterprise/platform/frontend:v26.2.2 platform: linux/amd64 environment: NGINX_LISTEN_PORT: 8001 # If not defined, defaults to 8000 networks: - frontend ports: - 8081:8001 # Map host port 8081 to container port 8001 restart: always depends_on: - backend ``` Customize the image with these environment variables: - `NGINX_LISTEN_PORT`: The port the NGINX process will listen on inside the container. Default: `8000`. - `NGINX_LISTEN_PORT_IPV6`: The NGINX listening port to open on the IPv6 address. Default: `8000`. - `NGINX_UPSTREAM_HOST`: The hostname of the backend service to which the NGINX process will route requests. Default: `backend`. - `NGINX_UPSTREAM_PORT`: The port where the backend service is exposed. Default: `8080`. If further customization of the config file is needed, mount a config map/secret over the templated NGINX configuration file at `/etc/nginx/templates/tower.conf.template`. See [SSL/TLS](./configuration/ssl_tls#configure-seqera-to-present-a-ssltls-certificate) for an example. ## Optional features ### Pipeline optimization Seqera Platform offers a service that optimizes pipeline resource requests. Refer to [Pipeline optimization](./configuration/pipeline_optimization.md) for more information. ### Studios [Studios](../studios/overview) is an interactive analysis environment available in organizational workspaces. To enable Studios, see [Studios deployment](./install-studios). :::note Studios is available from Seqera Platform v24.1. If you experience any problems during the deployment process please contact your account executive. Studios in Enterprise is not installed by default. ::: --- ## Platform: Helm [Helm](https://helm.sh) is an open-source command line tool used for managing Kubernetes applications. Seqera offers a [Helm chart](https://github.com/seqeralabs/helm-charts/tree/platform-1.0.7/charts/platform) to deploy Seqera Platform Enterprise on a Kubernetes cluster. :::info Prerequisites Other than the basic requirements [already listed in the Platform installation overview](./install-platform#prerequisites), you will need: - A Kubernetes cluster, preferably Amazon EKS - [Helm v3](https://helm.sh/docs/intro/install) and [kubectl](https://kubernetes.io/docs/tasks/tools/) installed locally ::: ## Installing the Helm chart Helm bundles resource definitions into templates for repeatable deployments: inputs can be passed to the Helm chart either via a YAML file or inline to replace the default values. The `values.yaml` file defines a chart's settings, such as container image tags, CPU/memory limits, ingress definition to expose the service, environment variables, etc. Each Helm chart generally comes with its own `values.yaml` file containing default settings, which can be overridden by providing a custom values file. More details about values customization can be found in the [Helm documentation](https://helm.sh/docs/topics/charts#values-files). These are chart-level settings. To set an application-level option, use the dedicated chart value where one exists, such as `platform.contactEmail` or `platform.smtp.host`, or set the matching environment variable with `backend.extraEnvVars` and `cron.extraEnvVars`. These apply per deployment, so set both where the backend and cron pods need the same value. The chart writes the `platform.YAMLConfigFileContent` field to the `tower.yml` file that Platform reads. Reserve it for structured values that no environment variable can express, such as a [custom navigation menu](./configuration/overview#custom-navigation-menu). For the available options, see [Configuration](./configuration/overview). 1. Fetch the default `values.yaml` file to customize the installation with your specific configuration: ```bash helm show values oci://public.cr.seqera.io/charts/platform --version 1.0.7 > my-values.yaml ``` Now edit the `my-values.yaml` file to set your options, such as internal container image registry, database connection details, license information, and other settings. To keep the file concise, drop the lines you don't want to customize and include only the settings you want to change. This makes your configuration easier to maintain. Values you don't specify fall back to the defaults defined in the chart's `values.yaml` file. For an example of a minimal configuration file, see the [example values file](https://github.com/seqeralabs/helm-charts/blob/platform-1.0.7/charts/platform/examples/kustomize/values.yaml). You can browse all the available configuration options in a tabular format in the [README](https://github.com/seqeralabs/helm-charts/tree/platform-1.0.7/charts/platform) file. 1. Install the chart from the public OCI registry in your desired namespace and with the values file customized in the previous step: ```bash helm install my-release oci://public.cr.seqera.io/charts/platform \ --version 1.0.7 \ --namespace my-namespace \ --create-namespace \ --values my-values.yaml ``` The chart will fail to install if mandatory values are not provided. ### Installing a Helm chart with Kustomize You can also use Kustomize to manage Helm chart installations. Kustomize provides further customization options. To install the Seqera Platform Enterprise Helm chart with Kustomize, see the [Kustomize example directory](https://github.com/seqeralabs/helm-charts/tree/platform-1.0.7/charts/platform/examples/kustomize). ## Upgrading the Helm chart To upgrade an existing Seqera Platform Enterprise Helm chart installation to a new version, run the following command, replacing `my-release` and `my-namespace` with your release name and namespace: ```bash helm upgrade my-release oci://public.cr.seqera.io/charts/platform \ --version NEW_VERSION \ --namespace my-namespace \ --values my-values.yaml ``` ## Uninstalling the Helm chart To uninstall the Seqera Platform Enterprise Helm chart, run the following command, replacing `my-release` and `my-namespace` with your release name and namespace: ```bash helm uninstall my-release -n my-namespace ``` --- ## Platform: Kubernetes Kubernetes deployments are recommended for production workloads requiring high availability and scalability. :::info Prerequisites Other than the basic requirements [already listed in the Platform installation overview](./install-platform#prerequisites), you will need: - A Kubernetes cluster, preferably Amazon EKS - [kubectl](https://kubernetes.io/docs/tasks/tools/) installed locally ::: ### Recommended resources | Component | CPU | Memory | | :---------- | :----- | :------------------------ | | Backend pod | 1 core | 4000 Mi request and limit | ## Container images Seqera Enterprise container images are hosted on a private registry (`cr.seqera.io`). Access is provided as part of your purchase. Contact [support](https://support.seqera.io) if you require access. We recommend mirroring these images to your own private container registry for production use. See [Mirroring container images](./configuration/mirroring) for details. For development and proof of concept installations, you can use [image pull secrets](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) to pull directly from the Seqera registry. ## Database configuration Create a MySQL database and user for Seqera: ```sql CREATE DATABASE tower; CREATE USER 'tower'@'%' IDENTIFIED BY 'your_secure_password'; GRANT ALL PRIVILEGES ON tower.* TO 'tower'@'%'; ``` See [Database configuration](./configuration/overview#seqera-and-redis-databases) for details. ## Redis or Valkey configuration Seqera Platform requires a Redis-compatible cache store for transient data, primarily Nextflow job metrics reporting. Both Redis and Valkey are supported. ### Supported versions | Cache / version | Status | | --------------- | ---------------------------- | | Redis 6.x | Not supported (from 26.1) | | Redis 7.2 | Supported | | Redis 7.4 | Supported | | Valkey 7.x | Supported (from 26.1) | :::note Managed cache services can offer newer engine versions than Seqera Platform is tested against. For example, Amazon ElastiCache offers Valkey up to 9.x. Select a version from the table. Newer major versions are not tested or supported. ::: ### Connection URL Configure the connection URL in your Seqera environment using the scheme that matches your cache backend: | Backend | Scheme | Example | | --------------- | ------------ | ---------------------------------------- | | Redis | `redis://` | `TOWER_REDIS_URL=redis://:6379` | | Redis with TLS | `rediss://` | `TOWER_REDIS_URL=rediss://:6380` | Seqera Platform 26.1 and later supports the Valkey 7 dial schema. No further configuration is required. Redis password and ACL configuration carry over unchanged when you migrate to Valkey. ### Managed service options Use a managed cache service for production: - [Amazon ElastiCache](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/WhatIs.html) (`cache.m4.large` or larger) - [Azure Managed Redis](https://learn.microsoft.com/azure/redis/overview) (production-capable tier appropriate for your workload) - [Google Memorystore](https://cloud.google.com/memorystore/docs/redis) (M2 tier or larger) :::caution Microsoft is retiring Azure Cache for Redis. As of April 1, 2026, new customers cannot create instances, and from October 1, 2026, no new instances can be created. For new Azure deployments, use [Azure Managed Redis](https://learn.microsoft.com/azure/redis/overview). ::: For migration guidance from Redis to Valkey on an existing installation, see [Cache layer changes](./upgrade#cache-layer-changes-redis-eol-and-valkey-support). ## Deploy Seqera Enterprise ### Create a namespace Create a namespace for Seqera resources: ```bash kubectl create namespace seqera-platform kubectl config set-context --current --namespace=seqera-platform ``` ### Seqera ConfigMap Download and configure a [ConfigMap](_templates/k8s/configmap.yml). See [Configuration](./configuration/overview.mdx) for more information. Deploy the ConfigMap to your cluster after it is configured: ```bash kubectl apply -f configmap.yml ``` :::note The `configmap.yml` manifest includes both the `tower.env` and `tower.yml` files. These files are made available to the other containers through volume mounts. ::: ### Seqera cron service Download the [cron service manifest](_templates/k8s/tower-cron.yml) file. To deploy the manifest to your cluster, run the following: ```bash kubectl apply -f tower-cron.yml ``` :::caution This container creates the required database schema the first time it instantiates. This process can take a few minutes to complete and must finish before you instantiate the Seqera backend. Ensure this container is in the `READY` state before proceeding to the next step. ::: ### Seqera frontend and backend Download the [manifest](_templates/k8s/tower-svc.yml). To deploy the manifest to your cluster, run the following: ```bash kubectl apply -f tower-svc.yml ``` #### Seqera frontend unprivileged From Seqera Platform Enterprise 26.2, Seqera publishes one frontend image, `cr.seqera.io/enterprise/platform/frontend:`. The image is unprivileged. It listens on an unprivileged port and doesn't run as the root user. Seqera no longer publishes the `-root` and `-unprivileged` tag variants. A manifest that references one fails to pull. The `tower-svc.yml` manifest earlier on this page is already configured for this image. If you maintain your own manifests, set the container port and the service `targetPort` to `8000`: ```yaml --- apiVersion: apps/v1 kind: Deployment metadata: name: frontend labels: app: frontend spec: ... containers: - name: frontend image: cr.seqera.io/enterprise/platform/frontend:v26.2.2 ports: - containerPort: 8000 env: - name: NGINX_LISTEN_PORT # If not defined, defaults to 8000. value: 8000 --- apiVersion: v1 kind: Service metadata: name: frontend spec: ports: - port: 80 targetPort: 8000 ``` The external `port` of the `frontend` service is independent of `NGINX_LISTEN_PORT`. Leave the `port` at `80` and change only the `targetPort` to match the port NGINX listens on inside the container: - If `NGINX_LISTEN_PORT` is unset, set `targetPort` to `8000`. - If you set `NGINX_LISTEN_PORT` to another value, set `targetPort` to that value. `NGINX_UPSTREAM_PORT` (default `8080`) sets the backend port that NGINX routes requests to, not the frontend listening port. Customize the image with these environment variables: - `NGINX_LISTEN_PORT` (default `8000`): The port the NGINX process will listen on inside the container. - `NGINX_LISTEN_PORT_IPV6` (default `8000`): The NGINX listening port to open on the IPv6 address. - `NGINX_UPSTREAM_HOST` (default `backend`): The hostname of the backend service to which the NGINX process will route requests. - `NGINX_UPSTREAM_PORT` (default `8080`): The port where the backend service is exposed. If further customization of the config file is needed, mount a config map/secret over the templated NGINX configuration file at `/etc/nginx/templates/tower.conf.template`. See [SSL/TLS](./configuration/ssl_tls#configure-seqera-to-present-a-ssltls-certificate) for an example. ### Seqera ingress An ingress is used to make Seqera Enterprise publicly accessible, load-balance traffic, terminate TLS, and offer name-based virtual hosting. The included ingress manifest will create an external IP address and forward HTTP traffic to the Seqera frontend. Download and configure the appropriate manifest for your infrastructure: - [Amazon EKS](_templates/k8s/ingress.eks.yml) - [Azure AKS](_templates/k8s/ingress.aks.yml) - [Google Kubernetes Engine](_templates/k8s/ingress.gke.yml) To deploy the manifest to your cluster, run the following: ```bash kubectl apply -f ingress.*.yml ``` See [Kubernetes ingress][k8s-ingress] for more information. If you don't need to make Seqera externally accessible, use a service resource to expose a [node port][k8s-node-port] or a [load balancer][k8s-load-balancer] service to make it accessible within your intranet. See the cloud provider documentation for configuring an ingress service on each cloud provider: - [Amazon][aws-configure-ingress] - [Azure][azure-configure-ingress] - [Google Cloud][google-configure-ingress] ### Check status Check that all services are up and running: ```bash kubectl get pods ``` ### Test the application See [Test deployment](./testing). ## Optional features ### Pipeline optimization Seqera Platform offers a service that optimizes pipeline resource requests. Refer to [Pipeline optimization](./configuration/pipeline_optimization.md) for more information. ### Studios [Studios](../studios/overview) is an interactive analysis environment available in organizational workspaces. To enable Studios, see [Studios deployment](./install-studios). :::note Studios is available from Seqera Platform v24.1. If you experience any problems during the deployment process [contact Seqera support](https://support.seqera.io). Studios in Enterprise is not installed by default. ::: ### High availability To configure Seqera Enterprise for high availability, note that: - The `backend` service can be run in multiple replicas - The `frontend` service is replicable, however in most scenarios it is not necessary - The `cron` service may only have a single instance - The `groundswell` service may only have a single instance [aws-configure-ingress]: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/ingress/annotations/ [azure-configure-ingress]: https://docs.microsoft.com/en-us/azure/application-gateway/ingress-controller-annotations [google-configure-ingress]: https://cloud.google.com/kubernetes-engine/docs/concepts/ingress [k8s-ingress]: https://kubernetes.io/docs/concepts/services-networking/ingress/ [k8s-load-balancer]: https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer [k8s-node-port]: https://kubernetes.io/docs/concepts/services-networking/service/#nodeport --- ## Studios: Docker Compose This guide describes how to deploy Studios for Seqera Platform Enterprise with Docker Compose. :::info Prerequisites Other than the basic requirements [already listed in the Studios installation overview](./install-studios#prerequisites), you will need: - Docker Engine and Docker Compose ::: ## Procedure 1. Create a folder for Connect metadata: ```bash mkdir -p $HOME/.tower/connect chmod 777 $HOME/.tower/connect ``` 1. Download the Studios [environment configuration file](./_templates/docker/data-studios.env). 1. Create an initial OIDC registration token: ```bash oidc_registration_token=$(openssl rand -base64 32 | tr -d /=+ | cut -c -32) ``` 1. Generate an RSA public/private key pair: ```bash openssl genrsa -out private.pem 2048 openssl rsa -pubout -in private.pem -out public.pem ``` 1. Download the [data-studios-rsa.pem](./_templates/docker/data-studios-rsa.pem) file and replace its contents with the content of your private and public key files (private key on top, public key directly beneath it). Save as `data-studios-rsa.pem` in the same directory as your `docker-compose.yml`. 1. Open `docker-compose.yml` and uncomment the volume mount for the PEM key file for the `backend` and `cron` services: ```yaml volumes: - $PWD/tower.yml:/tower.yml - $PWD/data-studios-rsa.pem:/data-studios-rsa.pem ``` 1. Open `data-studios.env` and set the following: - Uncomment the `connect-proxy` and `connect-server` services. - `PLATFORM_URL`: The same value as `TOWER_SERVER_URL` (e.g., `https://platform.example.com/` or `https://example.com/`). - `CONNECT_PROXY_URL`: A URL for the connect proxy subdomain (e.g., `https://connect.example.com`). - `CONNECT_OIDC_CLIENT_REGISTRATION_TOKEN`: The same value as `oidc_registration_token`. 1. Open `tower.env` and set the following: - `TOWER_DATA_EXPLORER_ENABLED`: Set to `true`. - `TOWER_DATA_STUDIO_CONNECT_URL`: The URL of the Studios connect proxy (e.g., `https://connect.example.com/`). - `TOWER_OIDC_REGISTRATION_INITIAL_ACCESS_TOKEN`: The same value as `oidc_registration_token`. - `TOWER_OIDC_PEM_PATH`: The file path to the PEM certificate (e.g., `/data-studios-rsa.pem`). 1. From Platform v26.1, Studios is enabled by default on all workspaces. To enable Studios on specific workspaces only, set the `TOWER_DATA_STUDIO_ALLOWED_WORKSPACES` environment variable (e.g., `TOWER_DATA_STUDIO_ALLOWED_WORKSPACES="12345,67890"`) on the Platform backend containers. To disable Studios for all workspaces, set `TOWER_DATA_STUDIO_ALLOWED_WORKSPACES=""` (an empty string). 1. Start your Platform instance: ```bash docker compose up -d ``` 1. To confirm that Studios is available, log in to your Platform instance and navigate to an organizational workspace that has Studios enabled. The **Studios** tab is included with the available tabs. ## Tool configuration This guide assumes that all services will be run in the same container as the rest of your Seqera Platform services. If you were using Studios prior to GA (v25.1) please review the `tower.env` file and make sure you are using the latest version which includes a new variable `TOWER_DATA_STUDIO_TEMPLATES__TOOL`. This variable needs to be added to the default/Seqera-provided Studio templates: `TOWER_DATA_STUDIO_TEMPLATES__TOOL: ''` The `TEMPLATE_KEY` can be any string, but the `TOOL_NAME` has to be the template name (`jupyter`/`vscode`/`rstudio`/`xpra`). You can also check the current template configuration using `https://towerurl/api/studios/templates?workspaceId=`. The response should include the `TOOL` configuration and template name (`jupyter`/`vscode`/`rstudio`/`xpra`) - not `custom`. ## Next steps To enable SSH access for Studios, see [Studios: SSH configuration](./studios-ssh). --- ## Studios: Helm [Helm](https://helm.sh) is an open-source command line tool used for managing Kubernetes applications. Seqera offers a [Helm chart](https://github.com/seqeralabs/helm-charts/tree/studios-1.7.5/charts/platform/charts/studios) to deploy Studios Enterprise on a Kubernetes cluster. :::info Prerequisites Other than the basic requirements [already listed in the Studios installation overview](./install-studios#prerequisites), you will need: - A Kubernetes cluster - [Helm v3](https://helm.sh/docs/intro/install) and [kubectl](https://kubernetes.io/docs/tasks/tools/) installed locally ::: ## Installation as part of Seqera Platform Enterprise The Studios Helm chart has been designed as a sub-chart of the main Seqera Platform Enterprise Helm chart, but can optionally be installed independently like the Platform chart. To install Studios as part of your Seqera Platform Enterprise deployment, make sure the `studios.enabled` value in your custom Platform's `values.yaml` file is set to `true`: ```yaml studios: enabled: true ``` At the same time, configure the desired Studios options as described in the [Studios Helm chart documentation](https://github.com/seqeralabs/helm-charts/tree/studios-1.7.5/charts/platform/charts/studios), in particular the Studios service domain and the subdomains that it will use for incoming connections. Also refer to the [example](https://github.com/seqeralabs/helm-charts/tree/studios-1.7.5/charts/platform/examples/studios) provided in the Helm charts repository. Then, follow the instructions in the Seqera Platform Enterprise installation guide [using Helm](./platform-helm) to install or upgrade your Platform deployment with Studios. ## Next steps To enable SSH access for Studios, see [Studios: SSH configuration](./studios-ssh). --- ## Studios: Kubernetes This guide describes how to deploy Studios for Seqera Platform Enterprise on Kubernetes. :::info Prerequisites Other than the basic requirements [already listed in the Studios installation overview](./install-studios#prerequisites), you will need: - A Kubernetes cluster - [kubectl](https://kubernetes.io/docs/tasks/tools/) installed locally ::: ## Tool configuration This procedure describes how to configure Studios for Seqera Enterprise deployments in Kubernetes. If you were using Studios prior to GA (v25.1) please review the `configmap.yaml` file and make sure you are using the latest version which includes a new variable `TOWER_DATA_STUDIO_TEMPLATES__TOOL`. This variable needs to be added to the default/Seqera-provided Studio templates: `TOWER_DATA_STUDIO_TEMPLATES__TOOL: ''` The `TEMPLATE_KEY` can be any string, but the `TOOL_NAME` has to be the template name (`jupyter`/`vscode`/`rstudio`/`xpra`). You can also check the current template configuration using `https://towerurl/api/studios/templates?workspaceId=`. The response should include the `TOOL` configuration and template name (`jupyter`/`vscode`/`rstudio`/`xpra`) - not `custom`. ## Procedure 1. Download the Kubernetes manifests for the Studios service: - [Proxy](./_templates/k8s/data_studios/proxy.yml) - [Server](./_templates/k8s/data_studios/server.yml) 1. Change your Kubernetes context to the namespace where your Platform instance runs: ```bash kubectl config set-context --current --namespace= ``` 1. Edit the `server.yml` file and set the `CONNECT_REDIS_ADDRESS` environment variable to the hostname or IP address of the Redis server configured for Platform. 1. Create an initial OIDC registration token, which can be any secure random string. For example, using openssl: ```bash oidc_registration_token=$(openssl rand -base64 32 | tr -d /=+ | cut -c -32) ``` 1. Edit the `proxy.yml` file and set the following variables: - `CONNECT_REDIS_ADDRESS`: The hostname or IP address of the Redis server configured for Seqera. - `CONNECT_PROXY_URL`: A URL for the connect proxy subdomain (e.g., `https://connect.example.com`). - `PLATFORM_URL`: The base URL for your installation (e.g., `https://platform.example.com/` or `https://example.com/`). - `CONNECT_OIDC_CLIENT_REGISTRATION_TOKEN`: The same value as the `oidc_registration_token` value created previously. 1. Edit the `ingress..yml` file appropriate for your Kubernetes environment: - Uncomment the `host` section at the bottom of the file. - Replace `` with the base domain of your installation. :::note In the case you're using AWS EKS, this assumes that you have an existing Seqera ingress already configured with the following fields: - `alb.ingress.kubernetes.io/certificate-arn`: The ARN of a wildcard TLS certificate that secures the Platform URL and connect proxy URL. For example, if `TOWER_SERVER_URL=https://example.com` and `CONNECT_PROXY_URL=https://connect.example.com`, the certificate must secure `example.com`, and `*.example.com` at the same time; otherwise, you may need to create a second ingress resource specifically for Studios. ::: 1. Generate an RSA public/private key pair. A key size of at least 2048 bits is recommended. In the following example, the `openssl` command is used to generate the key pair: ```bash openssl genrsa -out private.pem 2048 openssl rsa -pubout -in private.pem -out public.pem ``` 1. Download the [data-studios-rsa.pem](./_templates/docker/data-studios-rsa.pem) file and replace its contents with the content of your private and public key files created in the previous step, in the same order (private key on top, public key directly beneath it). 1. Apply a base64 encoding to the PEM file: ```bash base64_pem=$(cat data-studios-rsa.pem | base64 -w0) ``` 1. Create a secret file named `secret.yml`: ```yaml apiVersion: v1 kind: Secret metadata: name: platform-oidc-certs namespace: platform-stage data: oidc.pem: ``` 1. Create the secret: ```bash kubectl apply -f secret.yml ``` 1. Edit the `tower-svc.yml` file and uncomment the `volumes.cert-volume`, `volumeMounts.cert-volume`, and `env.TOWER_OIDC_PEM_PATH` fields. 1. Edit the ConfigMap named `platform-backend-cfg` in the `configmap.yml` by changing the following environment variables: - `TOWER_DATA_STUDIO_CONNECT_URL`: The URL of the Studios connect proxy, such as `https://connect.example.com/`. - `TOWER_OIDC_REGISTRATION_INITIAL_ACCESS_TOKEN`: The same value as the `oidc_registration_token` value created previously. 1. From Platform v26.1, Studios is enabled by default on all workspaces. To enable Studios on specific workspaces only, set the `TOWER_DATA_STUDIO_ALLOWED_WORKSPACES` environment variable (e.g., `TOWER_DATA_STUDIO_ALLOWED_WORKSPACES="12345,67890"`) on the Platform backend containers. To disable Studios for all workspaces, set `TOWER_DATA_STUDIO_ALLOWED_WORKSPACES=""` (an empty string). 1. Apply the updated configuration: ```bash kubectl apply -f configmap.yml ``` 1. Apply the configuration change to Platform: ```bash kubectl apply -f tower-svc.yml ``` 1. Restart the cron service of your deployment to load the updated configuration. For example: ```bash kubectl rollout restart deployment/cron ``` 1. Restart the Platform backend service of your deployment to load the updated configuration. For example: ```bash kubectl rollout restart deployment/backend ``` 1. Apply the updated ingress file and the Studios manifests: ```bash kubectl apply -f ingress..yml proxy.yml server.yml ``` 1. To confirm that Studios is available, log into Seqera and navigate to an organizational workspace that has Studios enabled. The **Studios** tab should be displayed in the sidebar. ## Next steps To enable SSH access for Studios, see [Studios: SSH configuration](./studios-ssh). --- ## Configure a private certificate authority for Studios Studio sessions open outbound TLS connections to the Connect proxy and to Seqera Platform. In a private network, those endpoints often present certificates issued by an internal certificate authority (CA) rather than a publicly trusted one. By default, a Studio session trusts only the public CAs in its container image's system trust store. As a result, the session fails to establish its tunnel and never reaches **running** status. Configure a custom CA to add your organization's CA to the trust store of every Studio session. This is a Seqera Platform Enterprise deployment setting, available in Enterprise only. This also applies if your organization inspects HTTPS traffic at the network boundary, because interception presents a certificate issued by an internal CA. :::info[**Prerequisites**] You need the following: - [Connect client](../studios/connect) version `0.13.0` or later. Earlier clients have no custom CA support. - Studio container images rebuilt against a `0.13.0` or later client. - Access to the Seqera Platform deployment configuration. - The root CA certificate in PEM format. ::: ## How Platform provisions the CA You supply the CA once, at the deployment level. Platform distributes it to sessions automatically. There is no per-workspace or per-compute-environment step. 1. Mount your CA certificate into the Platform container and set `TOWER_SSL_CUSTOM_CA_CERT_FILE` to its path. 2. Platform reads the file, validates that it contains a PEM certificate, and base64-encodes it. This happens once per Platform process, the first time a Studio session launches. 3. When a Studio session launches, Platform passes the encoded certificate to the session as `TOWER_CONNECT_CA_CERT_BASE64`. 4. The Connect client installs the certificate at the operating system level inside the session container. Installing the certificate at the OS level makes it available to the tunnel connection, the Connect client, the interactive tool, Fusion, and the AWS SDK. This setting covers Studio sessions only. It doesn't add your CA to Platform's own Java trust store. You configure trust for infrastructure that Platform itself reaches, such as private Git repositories, separately. See [SSL/TLS](./configuration/ssl_tls). Platform sets `TOWER_CONNECT_CA_CERT_BASE64` on every session, across all compute platforms. Users cannot override it. If a Studio's environment variables include `TOWER_CONNECT_CA_CERT_BASE64`, Platform removes the user-supplied value before launch. ## Configuration settings | Environment variable | Set by | Description | | :------------------- | :----- | :---------- | | `TOWER_SSL_CUSTOM_CA_CERT_FILE` | Administrator | Path to a PEM file containing your internal CA certificate, mounted into the Platform container. Unset by default, which disables custom CA provisioning. | | `TOWER_CONNECT_CA_CERT_BASE64` | Platform | The base64-encoded CA that the Connect client installs in the session. Platform derives this from `TOWER_SSL_CUSTOM_CA_CERT_FILE`. Do not set it by hand. | | `TOWER_CONNECT_CA_KEEP_DEFAULT` | Connect client | Whether the session keeps the container image's public CAs alongside your CA. Defaults to `true`. Platform does not set this variable, and the Studio form rejects environment variable names that begin with `TOWER_`. You cannot change the trust model yourself. Contact your Seqera account executive. See [Choose a trust model](#choose-a-trust-model). | ## Supply the CA to Platform Provide the certificate to the Platform container, then point `TOWER_SSL_CUSTOM_CA_CERT_FILE` at it. For a Kubernetes deployment, store the certificate in a `ConfigMap` and mount it: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: seqera-custom-ca data: internal-ca.pem: | -----BEGIN CERTIFICATE----- MIIDdzCCAl+gAwIBAgIEAgAAuTANBgkqhkiG9w0BAQUFADBaMQswCQYDVQQGEwJJ ... -----END CERTIFICATE----- ``` Mount the `ConfigMap` into the Platform container and set the environment variable to the mounted path: ```yaml env: - name: TOWER_SSL_CUSTOM_CA_CERT_FILE value: /etc/seqera/certs/internal-ca.pem volumeMounts: - name: custom-ca mountPath: /etc/seqera/certs readOnly: true volumes: - name: custom-ca configMap: name: seqera-custom-ca ``` For a Docker Compose deployment, mount the certificate as a volume and set the variable in your environment file: ```bash TOWER_SSL_CUSTOM_CA_CERT_FILE=/etc/seqera/certs/internal-ca.pem ``` Restart Platform to apply the change. Platform reads the certificate only at startup. :::note Supply the **root** CA certificate. Platform forwards the encoded certificate in the session launch environment. On AWS cloud compute environments, that environment has a size limit shared with everything else Platform injects. A single root certificate is well within that limit. A large chain or full bundle might not be. ::: Restart sessions to pick up the certificate. Running sessions are unaffected. ## Choose a trust model `TOWER_CONNECT_CA_KEEP_DEFAULT` controls whether your CA is added to the image's public CAs or replaces them. | Value | Trust anchors in the session | When to use | | :---- | :--------------------------- | :---------- | | `true` (default) | Your CA and the image's public CAs | Private routing without TLS interception, where some endpoints, for example object storage, still present publicly trusted certificates. | | `false` | Your CA only | A fully private network where an egress proxy re-signs all traffic with your internal CA. | :::warning Setting `TOWER_CONNECT_CA_KEEP_DEFAULT=false` breaks connections to any endpoint that presents a publicly trusted certificate. Set it to `false` only when your internal CA signs the certificate for every endpoint a session reaches, including object storage and package indexes. ::: Full closure applies to the Connect client and Fusion. It does not extend to the interactive tool or to Node-based tooling, both of which keep the container image's public roots. Treat `false` as a way to force internal traffic through your CA, not as a guarantee that nothing in the session can reach a publicly trusted endpoint. ## What the CA reaches Most tooling in a session reads the operating system trust store and is covered automatically. The session points runtimes that ship their own certificate list at the same certificate: | Consumer | How it picks up the CA | | :------- | :--------------------- | | `curl`, `wget`, `git`, `openssl`, Python's standard library | The session's system trust store. | | Go programs, including `dockerd` | The system trust store. Registry pulls performed by `dockerd` work. | | Node tooling, including the VS Code extension marketplace client | Added to Node's built-in roots. Node's roots are compiled into the binary. Your CA can be added to them but cannot replace them. | | Python `requests` and `pip` | Pointed at the same certificate instead of their bundled store. | | Java, including Nextflow | A generated keystore is mounted over the runtime's default trust store. Because no environment variable is involved, this works with Nextflow, whose launcher discards `JAVA_TOOL_OPTIONS`. | Studio sessions run Linux containers, and the mechanism relies on standard Linux trust store paths and TLS environment variables. ## Limitations - **Java runtimes installed mid-session.** The session discovers Java runtimes once, when it starts. If you install a JDK during a session, for example with `micromamba install openjdk nextflow`, that session does not pick up the CA, and Java TLS to internal endpoints fails. Because the session checkpoint captures the install, stopping and starting the session resolves it. Where possible, use a Studio image that already includes the Java runtime you need. - **Nested containers.** Containers launched from inside a session with `dockerd` don't inherit the CA, because each has its own image filesystem and environment. To give a nested container the CA, add it to that container's image or mount it in at `docker run` time. - **Full closure doesn't reach every process.** See [Choose a trust model](#choose-a-trust-model). - **Browser trust is separate.** See [Browser trust is separate](#browser-trust-is-separate). :::note Contact your Seqera account executive if a session in a private network behaves differently from what's described here. ::: ## Browser trust is separate Configuring the CA in Platform lets the *session* establish its outbound connections. It does not affect the user's browser. To open a Studio whose Connect endpoint uses an internal certificate, the internal CA must also be present in the trust store of the user's own machine or browser. Distributing the CA to users is a separate decision from this setting. ## Troubleshoot a private CA configuration [Studios troubleshooting](../troubleshooting_and_faqs/studios_troubleshooting) covers the symptoms a Studio user sees. The following are errors you're likely to meet while configuring the CA. #### Error: `Custom CA certificate file not found or not readable` Studio sessions fail to launch and Platform logs the configured path. This issue occurs when `TOWER_SSL_CUSTOM_CA_CERT_FILE` points at a path that does not exist in the Platform container, or that the Platform process cannot read. Platform itself starts normally. The error surfaces the first time a Studio session launches. To resolve, confirm the volume or `ConfigMap` is mounted at the path the variable names, and that the file is readable by the Platform user. #### Error: `Custom CA certificate file does not contain a PEM certificate` Studio sessions fail to launch after Platform finds the file. This issue occurs when the file is not PEM-encoded. For example, the file is a DER or PKCS#12 certificate, or a key file supplied by mistake. As with the previous error, Platform starts normally and the error surfaces at the first session launch. To resolve, convert the certificate to PEM so that it contains a `BEGIN CERTIFICATE` block, then restart Platform. #### Error: `x509: certificate signed by unknown authority` A session fails to establish its tunnel and does not reach **running** status. This issue occurs when the Connect client does not trust the certificate the endpoint presents. Check the following: 1. The Studio image runs a Connect client of version `0.13.0` or later. Earlier clients ignore the provisioned certificate. 2. `TOWER_SSL_CUSTOM_CA_CERT_FILE` is set, and you restarted Platform after you set it. 3. The certificate supplied is the CA that issued the endpoint's certificate. If the client is older than `0.13.0` and you cannot update it, build a custom Studio image with your CA added to the image's trust store. See [Custom container images](../studios/container-images). #### Error: `PKIX path building failed` A Java process inside the session can't verify a certificate that the rest of the session trusts. The process is commonly Nextflow reporting to Platform with `-with-tower`, or reaching an internal Git server or S3-compatible storage. Public endpoints keep working, because Java ships its own public roots. This issue occurs when the Java runtime was installed after the session started. The runtime wasn't present when the session configured Java trust. To resolve, stop and start the session. Because the session checkpoint captures the installed runtime, the runtime is present the next time the session configures trust. Where possible, use a Studio image that already includes the Java runtime you need. If the error persists on a restarted session, the runtime is in a layout the session didn't recognize. As a workaround, import the CA into that runtime's default keystore. Run the import from a startup script so that it's re-applied each session, rather than frozen into the checkpoint when your CA rotates: ```bash keytool -delete -alias connect-custom-ca -keystore "$JAVA_HOME/lib/security/cacerts" \ -storepass changeit 2>/dev/null || true keytool -importcert -trustcacerts -noprompt -alias connect-custom-ca \ -keystore "$JAVA_HOME/lib/security/cacerts" -storepass changeit -file /path/to/ca.pem ``` Don't set `JAVA_TOOL_OPTIONS` for Nextflow. Its launcher clears that variable, and Nextflow silently ignores the setting. Use `NXF_OPTS` if you prefer a separate trust store over importing into the default one. #### Error: `SELF_SIGNED_CERT_IN_CHAIN` A Node-based tool inside the session, such as the VS Code extension marketplace client, rejects a certificate issued by your CA. Because Node's trusted roots are compiled into the binary, the session adds your CA through the `NODE_EXTRA_CA_CERTS` environment variable rather than through the system trust store. This issue occurs when a tool doesn't read that variable, or when the tool is configured to use its own proxy or certificate settings. To resolve, confirm the tool honors `NODE_EXTRA_CA_CERTS`, and check whether a tool-specific proxy or certificate setting overrides it. Contact your Seqera account executive if a Node-based tool in a session can't reach an internal endpoint. --- ## Studios SSH configuration (public preview) :::warning[Advanced configuration] Enabling SSH for running Studio sessions requires changes to platform configuration and therefore careful consideration of security and networking. This feature requires network-level changes to your infrastructure that may need coordination with your infrastructure or networking team. Improperly configured SSH access may introduce security vulnerabilities. ::: SSH access enables direct terminal connections to running Studio sessions using standard SSH clients, supporting [VS Code Remote SSH](https://code.visualstudio.com/docs/remote/ssh) and terminal access. ## Version requirements To enable SSH access to running Studio sessions, you need: - **Seqera Platform**: Version 25.3.3 or later - **Connect server and proxy**: Version 0.12.0 or later - **Connect client**: Version 0.12.0 or later If you have pinned Studio container images to specific versions, you will need to [migrate](../studios/managing#migrate-a-studio-from-an-earlier-container-image-template) them to the required Connect client version. :::info Prerequisites - Studios enabled (see [Studios installation](./install-studios)) - Access to Platform configuration - Administrative access to modify your deployment infrastructure - Ability to configure network routing and firewall rules ::: ## Requirements overview To enable SSH access for Studios you need: 1. **SSH key pair**: A key pair for the connect-proxy to authenticate to a Studio 2. **Platform configuration**: Environment variables to enable and configure SSH on the Platform backend pods 3. **Proxy configuration**: Environment variables to enable SSH on the connect-proxy 4. **Network configuration**: Layer 4 (TCP) access to the SSH port on the connect-proxy, typically via a dedicated service that separates SSH traffic from HTTPS traffic :::note Configuration variables are prefixed by service: `TOWER_` for Platform backend, `CONNECT_` for connect-proxy. ::: ## Step 1: Generate SSH key pair SSH access to Studios requires a dedicated SSH key pair that establishes trust between the connect-proxy and Studio containers. This key pair serves two purposes: 1. **Authentication**: The connect-proxy uses the private key to authenticate to SSH servers running in Studio containers 2. **Security validation**: Studio SSH Server can verify that SSH connections originate from your authorized connect-proxy rather than external sources Generate an SSH key pair for the connect-proxy. ```bash # Generate key ssh-keygen -t ed25519 -C "connect-proxy" -f /path/to/connect-proxy-key # Generate fingerprint ssh-keygen -lf /path/to/connect-proxy-key ``` Platform uses the fingerprint output (SHA256 hash) in configuration to enable an additional security layer. After you configure this fingerprint, Studio SSH servers only accept connections from clients presenting this specific key, preventing unauthorized SSH access. :::warning[Key consistency] All Connect proxy instances must use a **shared SSH key** to prevent host key verification errors. In high availability deployments, ensure all replicas access the shared key. ::: ### Key distribution The SSH private key must be securely stored and accessible to all connect-proxy service instances. The method for distributing and mounting the key depends on your deployment platform's secrets management capabilities. ## Step 2: Configure Platform Add the following environment variables to your Platform backend configuration. All Platform configuration variables use the `TOWER_` prefix: ```yaml TOWER_DATA_STUDIO_CONNECT_SSH_KEY_FINGERPRINT: "SHA256:NEu6MAPGJpImFJ3raQzv6+NubCPy/92hqR+CVyMjKvM" TOWER_DATA_STUDIO_SSH_ALLOWED_WORKSPACES: "12345,67890" TOWER_SSH_KEYS_MANAGEMENT_ENABLED: "true" TOWER_DATA_STUDIO_CONNECT_SSH_PORT: "2222" TOWER_DATA_STUDIO_CONNECT_SSH_ADDRESS: "" ``` **Configuration details:** - `TOWER_DATA_STUDIO_CONNECT_SSH_KEY_FINGERPRINT`: SSH key fingerprint from Step 1. After you configure this fingerprint, Studio SSH servers only accept connections using this key. This rejects connections not originating from your connect-proxy (recommended for security). - `TOWER_DATA_STUDIO_SSH_ALLOWED_WORKSPACES`: Comma-separated workspace IDs allowed to use SSH. Set to an empty string to enable for all Platform Workspaces. Don't set the environment variable to disable the feature entirely. - `TOWER_SSH_KEYS_MANAGEMENT_ENABLED`: Set to `true` to enable SSH key management in Platform. - `TOWER_DATA_STUDIO_CONNECT_SSH_PORT`: SSH port (must match proxy configuration). - `TOWER_DATA_STUDIO_CONNECT_SSH_ADDRESS`: Set this when SSH traffic uses a different DNS address than the regular connect URL. Use this when SSH traffic needs to bypass Layer 7 load balancers used for HTTPS traffic. If not set, the regular connect URL is used for SSH. ## Step 3: Configure proxy Add the following configuration to the connect-proxy service. All connect-proxy configuration variables use the `CONNECT_` prefix: **Environment variables:** - `CONNECT_SSH_ENABLED`: Set to `true` - `CONNECT_SSH_ADDR`: Set to `:2222` (or your chosen SSH port) - `CONNECT_SSH_KEY_PATH`: Path to the SSH private key file (e.g., `/secrets/ssh-key`) **Volume mounts:** - Mount the SSH private key at the path specified in `CONNECT_SSH_KEY_PATH` - Ensure the key is mounted read-only for security The specific implementation depends on your deployment method (Kubernetes manifests, Helm values, Docker Compose, etc.). ## Step 4: Network access requirements SSH access requires the following networking configuration in your infrastructure: **Required:** - Layer 4 (TCP) network access to the connect-proxy on the configured SSH port (default: 2222) - DNS resolution to the connect-proxy SSH endpoint - Firewall rules permitting SSH traffic on the configured port **Recommended:** - Separate network routing for SSH traffic from HTTP/HTTPS traffic for security - If your existing load balancers operate at Layer 7 (HTTP/HTTPS), configure a separate endpoint for Layer 4 (TCP) SSH traffic and set `TOWER_DATA_STUDIO_CONNECT_SSH_ADDRESS` to this endpoint :::note Network configuration is specific to your infrastructure and deployment environment. This may require coordination with your infrastructure or networking team to ensure proper routing and security controls are in place. ::: ## Step 5: Apply configuration After configuring all the required settings: 1. Apply the updated Platform configuration and restart the Platform backend and cron services to load the new settings 2. Apply the updated connect-proxy configuration and restart the connect-proxy service 3. Implement the network access requirements from Step 4 Verify that all services restart successfully and the configuration changes are active. ## Verify SSH access 1. Ensure Studios is enabled for your workspace. 2. Add a Studio with **SSH Connection** enabled. 3. Start the Studio. 4. Test SSH connection: ```bash ssh @@ -p 2222 ``` For detailed usage instructions and VS Code setup, see [Connect to a Studio via SSH](../studios/managing#connect-to-a-studio-via-ssh-public-preview). ## Troubleshooting For SSH connection issues, see [Studios troubleshooting](../troubleshooting_and_faqs/studios_troubleshooting#ssh-connections-public-preview). --- ## Configure Studios data transfer quotas Studio sessions stream data between users and their interactive environments through the Connect proxy. Apply _data transfer quotas_ to cap how many bytes a user or client IP transfers over a set time window. Data transfer quotas are available in Seqera Platform Enterprise only. The Connect proxy (`connect-proxy`) enforces quotas. Quotas are opt-in. If you do not define a policy, the proxy does no counting or enforcement and adds no overhead. :::warning[Advanced configuration] You configure data transfer quotas through deployment settings that depend on your Redis backend and load-balancer networking. The `ip` bucket requires network-level changes that you might need to coordinate with your infrastructure or networking team. ::: :::info[**Prerequisites**] You need the following: - Connect server and proxy version `0.12.0` or later. - Studios enabled. See [Studios installation](./install-studios). - Access to the Connect proxy deployment configuration. - A Redis 7.0 or later backend with server-side scripting enabled. See [Redis requirements](#redis-requirements). - For the `ip` bucket, the ability to configure client-IP resolution on your load balancer. ::: ## Quota enforcement You define a _policy_ that groups traffic into _buckets_. A bucket is a logical counter keyed by an identity, such as a user ID or a client IP. Each bucket has one or more _quotas_, and each quota sets a byte limit over a time window. The proxy counts the bytes transferred in each direction (client to Studio and Studio to client) and totals them against the applicable buckets. When a bucket exceeds a quota, the proxy denies further traffic for that bucket until the window resets. Proxy pods reconcile their counts in a shared Redis instance. Quotas apply consistently regardless of which pod serves a request. ### Fixed time windows Each quota window is _fixed_. It starts on the first byte counted and expires a set duration later, regardless of activity in between. When a window expires, the counter resets, and the next transfer opens a fresh window. :::caution Because windows are independent, a user can transfer close to a full quota immediately before a window resets and another full quota immediately after. A single boundary therefore allows up to roughly twice the cap. Averaged over time, usage converges to the configured rate. Use shorter windows to tighten this bound. ::: ### Behavior when a bucket exceeds a quota When a bucket is over quota, the proxy stops the transfer. On a standard HTTP request, the user receives an `HTTP 429` (Too Many Requests) response. On an active streaming connection (WebSocket or SSH), the proxy tears down the connection. :::note Some interactive clients, including VS Code, might not recover cleanly after a denied transfer and require the user to reconnect. ::: ## Redis requirements Data transfer quotas require a Redis backend that exposes the `incrby`, `expire`, `ttl`, `mget`, `hincrby`, `hset`, `hsetnx`, `eval`, and `evalsha` commands. If you configure a policy and any of these commands is missing, renamed, or blocked by access control lists (ACLs), the proxy fails to start rather than enforce quotas incorrectly. :::caution Enforcement relies on server-side Lua scripting (`eval`/`evalsha`). A managed Redis service with scripting disabled cannot enforce quotas. Redis 7.0 or later is recommended for correct time-to-live (TTL) handling on quota counters. ::: ## Define a policy A policy is a JSON document that lists the buckets to track. Each bucket has a `name`, one or more `quotas`, and one or more `extractors` that bind a protocol to the value that identifies the bucket. ```json { "buckets": [ { "name": "user_id", "quotas": [ { "bytes": "1TB", "window": "720h", "on_exceed": "deny" } ], "extractors": [ { "protocol": "http", "source": { "type": "header", "name": "X-Connect-Sub" } }, { "protocol": "ssh", "source": { "type": "permissions", "field": "userId" } } ] }, { "name": "ip", "quotas": [ { "bytes": "50GB", "window": "24h", "on_exceed": "deny" }, { "bytes": "3GB", "window": "1h", "on_exceed": "deny" } ], "extractors": [ { "protocol": "any", "source": { "type": "client_ip" } } ] } ] } ``` ### Quota fields Each entry in a bucket's `quotas` array defines one limit: - `bytes` — The transfer limit. Use binary multipliers, where `KB` is 2^10 bytes, `MB` is 2^20, `GB` is 2^30, and `TB` is 2^40. The proxy reads a bare integer as bytes. - `window` — The time window, in Go duration syntax (for example, `1h`, `24h`, `720h`). - `on_exceed` — The action to take on breach. The proxy supports only `deny`. Quotas in the same bucket share the same byte counters. For example, you can combine a long-window fair-use ceiling with a short-window burst cap. ### Extractor source types Each extractor's `source.type` determines how the proxy identifies the bucket. The proxy supports only the following values: | `type` | Fields | Protocol | Identifies by | |--------|--------|----------|---------------| | `header` | `name` | HTTP | The value of the named HTTP header | | `permissions` | `field` | SSH | An entry from the SSH permissions data set at authentication | | `jwt` | `token`, `claim` | HTTP | A claim from the validated JSON Web Token (JWT) | | `client_ip` | _(none)_ | `any` | The proxy-resolved client IP | :::caution Use a `header` source only with a header that the proxy sets itself. The identity header `X-Connect-Sub` is safe because the proxy overwrites or strips any client-supplied value during authentication. Pointing a `header` source at a header the proxy does not control lets a user forge it and charge their traffic to another user's quota. ::: The proxy validates the policy at startup. Unknown source types, duplicate bucket names, two bindings for the same protocol in one bucket, or mixing `any` with a specific protocol all prevent startup. ## Apply the policy Configure quotas with the following environment variables. Set either `CONNECT_POLICY_B64` or `CONNECT_POLICY_FILE`, not both. | Environment variable | Default | Description | |----------------------|---------|-------------| | `CONNECT_POLICY_B64` | _(empty)_ | Base64-encoded JSON traffic policy. Empty or unset disables telemetry and quota enforcement. Mutually exclusive with `CONNECT_POLICY_FILE`. | | `CONNECT_POLICY_FILE` | _(empty)_ | Path to a JSON traffic policy file. Mutually exclusive with `CONNECT_POLICY_B64`. | | `CONNECT_TELEMETRY_FLUSH_INTERVAL` | `30s` | How often the proxy writes in-memory byte counters to Redis. | | `CONNECT_TELEMETRY_TTL` | `168h` | TTL for cumulative per-bucket telemetry keys in Redis (7 days). | | `CONNECT_TELEMETRY_STREAM_EMIT_INTERVAL` | `1s` | How often long-lived streams (WebSocket and SSH) report transferred bytes. | In a Kubernetes deployment, store the policy in a `ConfigMap` and inject it into the proxy configuration as base64. :::info The proxy reads the policy once at startup and does not reload it at runtime. To change a quota, update the policy `ConfigMap`, then perform a rolling restart of the proxy Deployment. Editing the `ConfigMap` alone has no effect on running pods. ::: ## Resolve the client IP for the `ip` bucket The `ip` bucket keys on the client IP as the proxy resolves it. Behind a load balancer, that resolution needs explicit configuration. If you skip this step, the proxy counts traffic against Kubernetes node IPs instead of client IPs. Because many users share the same nodes, one busy node can exceed the limit and deny traffic to unrelated users. A single user's traffic also fragments across nodes. :::note This section applies only to policies that use the `ip` bucket. A policy that uses only `user_id` quotas needs no load-balancer configuration. ::: The proxy resolves the client IP differently for HTTP and SSH traffic. Configure each separately. ### Trust `X-Forwarded-For` for HTTP traffic An HTTP(S) load balancer, for example an AWS Application Load Balancer, terminates the connection and appends the real client IP to `X-Forwarded-For`. Set `CONNECT_TRUSTED_PROXY_CIDRS` to the CIDR ranges of the hops between your load balancer and the proxy: ```bash CONNECT_TRUSTED_PROXY_CIDRS="10.0.0.0/8 172.16.0.0/12 192.168.0.0/16" ``` The proxy walks `X-Forwarded-For` from the right, skips trusted hops, and takes the first untrusted address as the client. - Set the CIDRs to the node or virtual private cloud (VPC) ranges between your load balancer and the proxy. Use whichever address the proxy sees as its immediate peer. - The default `127.0.0.1/32` is a deliberate no-op. If you leave it unset, the proxy trusts nothing and falls back to the socket peer. - Trust only as narrow a range as necessary. Any client whose address falls inside a trusted CIDR can forge `X-Forwarded-For` and charge its traffic to another IP's bucket. ### Preserve the source IP for SSH traffic SSH runs at Layer 4, through a network load balancer. Layer 4 carries no `X-Forwarded-For` header, and `CONNECT_TRUSTED_PROXY_CIDRS` has no effect. The real client IP must survive the network path instead. For a `NodePort` SSH service, set the following: ```yaml externalTrafficPolicy: Local ``` `Local` skips the kube-proxy source network address translation (NAT) and gives the pod the real client IP. As a trade-off, only nodes running a proxy pod stay healthy in the load balancer's target group. Configure the load balancer health check to probe the `NodePort` so that nodes without a proxy pod drop out of rotation. Alternatively, enable the PROXY protocol on the load balancer. Without one of these two options, SSH sessions bucket on node IPs. ### Confirm client-IP resolution Generate traffic and confirm the resolved address is a real client IP, not a node IP: - **HTTP** — At `debug` log level, the `telemetry http` log line shows `keys: ["ip:", ...]`, and the reverse-proxy log shows `client_ip`. Both should be the public client address. - **SSH** — The proxy logs each connection's `remote address`. The value should be the client IP, not a private node address such as `172.16`–`172.31.x`. If the `ip:` key (or `client_ip` / `remote address`) shows a private or node range (`10.x`, `172.16`–`172.31.x`, or `192.168.x`), resolution is not configured correctly. Revisit [Trust `X-Forwarded-For` for HTTP traffic](#trust-x-forwarded-for-for-http-traffic) and [Preserve the source IP for SSH traffic](#preserve-the-source-ip-for-ssh-traffic). :::note The proxy emits the per-request `telemetry http` lines at `debug` level only. They do not appear at the default `INFO` level. Set `CONNECT_LOG_LEVEL=debug` temporarily while you verify, then revert it. ::: ## Monitor data transfer ### Prometheus metrics When you load a policy, the proxy registers three metrics, labeled by bucket _name_ only (for example, `user_id` or `ip`) and never by the extracted value. Metric cardinality therefore stays bounded by your policy rather than by the number of users or IP addresses. Each configured bucket also gets a zero-valued series so that dashboards show a baseline before the first breach. | Metric | Type | Meaning | |--------|------|---------| | `connect_proxy_quota_exceeded_total{bucket}` | Counter | Number of times a bucket crossed a quota and the proxy denied traffic | | `connect_proxy_quota_cleared_total{bucket}` | Counter | Number of times a bucket returned under quota and the proxy resumed traffic | | `connect_proxy_quota_breached_keys{bucket}` | Gauge | Number of bucket keys currently over quota | :::note Metric scraping and dashboards are available in Seqera Platform Cloud only. For self-managed Enterprise deployments, use the Redis and log-based checks described later on this page to observe quotas. ::: ### Inspect counters in Redis The proxy stores quota data in Redis under the `connect:telemetry` prefix. For each bucket value, the proxy maintains a cumulative record for analytics and a separate counter per quota window for enforcement. The enforcement counters carry a `q` suffix, for example `:q60` for a 60-second window. Use the following commands to inspect a user's counters during troubleshooting: ```bash # List telemetry keys. Use SCAN, not KEYS — KEYS blocks the whole Redis server. SCAN 0 MATCH connect:telemetry:* COUNT 100 # A user's quota counter and time remaining in the window GET connect:telemetry:user_id:42:q60 TTL connect:telemetry:user_id:42:q60 # A user's cumulative transfer totals HGETALL connect:telemetry:user_id:42 ``` Interpret the quota counter as follows: - Value below the cap — the bucket is within quota. - Value at or above the cap — the bucket is breached, and the proxy denies traffic until the window expires. - Key absent (`TTL` returns `-2`, `GET` returns nil) — the window has expired, and the next transfer opens a new one. ### Log messages Watch for these proxy log lines: - `telemetry flushed {entries: N}` — a healthy flush to Redis. - `telemetry flush failed, will retry next tick` — a Redis write error. The proxy retains the counters and retries them. - `quota exceeded, denying traffic for bucket` / `quota cleared, resuming traffic` — enforcement transitions. ## Troubleshooting For quota enforcement and client-IP issues, see [Studios troubleshooting](../troubleshooting_and_faqs/studios_troubleshooting#data-transfer-quotas). --- ## Test deployment After your [Docker Compose](./platform-docker-compose) or [Kubernetes](./platform-kubernetes) installation is complete, follow these steps to test whether the application is running as expected: 1. Log in to the application. 2. Create an [organization](../orgs-and-teams/organizations). 3. Create a [workspace](../orgs-and-teams/workspace-management) within the organization. 4. Create a new [compute environment](../compute-envs/overview). 5. Add your [GitHub credentials](../git/overview). 6. Select **Quick Launch** from the **Launchpad** tab in your workspace. 7. Enter the repository URL for the `nf-core/rnaseq` pipeline (`https://github.com/nf-core/rnaseq`). 8. In the **Config profiles** drop-down, select the `test` profile. 9. In **Pipeline parameters**, change the output directory to a location based on your compute environment: ```yaml # Uncomment to save to an S3 bucket # outdir: s3:///results # Uncomment to save to a scratch directory (Kubernetes) # outdir: /scratch/results ``` 10. Select **Launch**. You'll be redirected to the **Runs** tab for the workflow. After a few minutes, progress logs will be listed in that workflow's **Execution log** tab. --- ## Upgrade deployment Upgrade your Seqera Platform Enterprise installation and database to version 26.2. The sections for earlier versions list the extra changes each upgrade path needs. :::info[**Prerequisites**] - Make a backup of your Platform database. - Complete each intermediate major version upgrade if you're upgrading from a version earlier than 25.1. For example, upgrade from 24.1 to 25.1, then to 26.1, then to 26.2. The sections below list the requirements for each version. - Make sure no pipelines are running during the upgrade. Data from active runs can be lost. - Make sure no Studios are running. Active Studios with mounted data can be irreparably damaged. ::: ## Upgrade from versions earlier than 24.1 - If you're upgrading from a version earlier than 23.4.1, upgrade your installation to version 23.4.4 **first**, before you upgrade to version 26.2 with the steps on this page. - **MySQL 8 required** From version 23.4, Seqera Enterprise supports only MySQL 8. If you run MySQL 5.6 or 5.7, upgrade your database to a supported MySQL version before you upgrade Seqera. See [Database changes](#database-changes) for the current baseline. ## Upgrade from versions 24.1–25.1 - **OIDC secrets injection changes** The `oidc-token-import` Micronaut environment replaces `auth-oidc-secrets`. If you use `auth-oidc-secrets`, change the `MICRONAUT_ENV` environment variable in your manifest during the upgrade. If you turn on the feature with the `TOWER_OIDC_TOKEN_IMPORT` environment variable, no change is needed. - **MariaDB driver: new MySQL connection parameter required** MariaDB driver 3.x requires the `permitMysqlScheme=true` parameter in the connection URL to connect to a MySQL database: `jdbc:mysql://:/tower?permitMysqlScheme=true` Update every deployment that uses a MySQL database, whatever the MySQL version, when you upgrade to version 24.1 or later. - **Redis version change and property deprecation** - From version 24.2, Seqera Enterprise requires Redis 6.2 or later. **From 26.1, Redis 6.x is not supported. See [Cache layer changes](#cache-layer-changes-redis-eol-and-valkey-support).** - From version 24.2, the `redisson.*` configuration properties are deprecated. If you set `redisson.*` properties directly: - Replace `/redisson/*` references in AWS Parameter Store entries with `TOWER_REDIS_*` environment variables. - Replace `redisson.*` references in `tower.yml` with `TOWER_REDIS_*` environment variables. - **Micronaut property key changes** In version 24.1, the property that sets the expiration time of the JWT access token changed. This token authenticates web sessions and the requests Nextflow sends to Seqera Platform. | Previous | New | | --- | --- | | `micronaut.security.token.jwt.generator.access-token.expiration` | `micronaut.security.token.generator.access-token.expiration` | If you customized this value, use the new property name. ## Upgrade from version 25.3.x to 26.1 You can upgrade directly from 25.3.x to 26.1. Review the following change, then follow the [general upgrade steps](#general-upgrade-steps). - **Secret key rotation** To configure [secret key rotation](https://docs.seqera.io/platform-enterprise/enterprise/configuration/overview#secret-key-rotation): - Before you turn on key rotation, back up your Platform database and your current crypto secret key to prevent data loss. - Configure the same previous and new secret key values on every backend pod or container in your deployment. - Start the Platform cron service only after every backend pod or container is ready and running. ## Upgrade from version 26.1.x to 26.2 You can upgrade directly from 26.1.x to 26.2. Before you upgrade, review the breaking changes and default changes later on this page, then follow the [general upgrade steps](#general-upgrade-steps). ## 26.2 breaking changes ### Audit log v1 writes removed Seqera Platform Enterprise 26.2 writes audit events only to the v2 schema. This is a **breaking change** for direct database consumers and custom ETL jobs that still read new events from the legacy v1 schema (`tw_audit_log` table). - The `TOWER_AUDIT_LOG_V2_WRITE_MODE` setting is removed and has no effect. Remove it from your configuration. - Platform writes no new rows to the v1 schema. Existing rows remain until the audit log retention period deletes them. While the table has records, they stay visible in the **Table v1** tab of the Admin panel **Audit logs** page. ## Database changes Seqera Platform Enterprise 26.1 changed the supported database versions. Before you upgrade, check your database against the following table. | Database / version | 26.x status | Action | | --- | --- | --- | | MySQL 5.7 | No longer tested or supported (upstream end of life) | Upgrade to MySQL 8.4 before upgrading to 26.1 | | MySQL 8.0 | No longer tested or supported (upstream end of life April 2026) | Upgrade to MySQL 8.4 | | MySQL 8.4 (LTS) | Recommended default | No action | | MariaDB | MariaDB driver 3.x | No action | | AWS Aurora MySQL (provisioned) | Supported | No action | | AWS Aurora Serverless | Not supported (existing guidance) | Migrate to a supported configuration | If you run MySQL 5.7 or 8.0, migrate your database **before** you upgrade the application to 26.1. The `migrate-db` container that Seqera supplies does not run against an unsupported database version. ## Cache layer changes: Redis EoL and Valkey support Seqera Platform Enterprise 26.1 added support for Valkey and raised the minimum Redis version. | Cache / version | 26.x status | Action | | --- | --- | --- | | Redis 6.x | Not supported from 26.1 | Upgrade to Redis 7.x or migrate to Valkey 7.x | | Redis 7.2 | Supported | No action | | Redis 7.4 | Supported | No action | | Valkey 7.x | Newly supported in 26.1 upwards | Optional migration path from Redis | :::note Redis 6.2 remains an upstream extended-support release until 1 April 2027, and Amazon ElastiCache supports Redis OSS 6 until 31 January 2027, with paid extended support until 31 January 2030. These upstream dates do not extend Seqera support. Seqera Platform 26.1 is not tested against Redis 6.x. Upgrade your cache before you upgrade Seqera Platform. Use Redis 7.2 or 7.4, or Valkey 7.x. Seqera does not test or support newer major versions. ::: ### Migrate from Redis to Valkey To migrate from Redis to Valkey, point `TOWER_REDIS_URL` at your Valkey 7.x installation. No other configuration is needed because Valkey 7.x supports the same schema as Redis. :::note Redis password and ACL configuration carry over unchanged when you migrate to Valkey. ::: ## Single unprivileged frontend image From 26.2, Seqera publishes one frontend image. It runs as a non-root user and was previously tagged `-unprivileged`. Seqera no longer publishes the `-root` tag variant or the `-unprivileged` tag alias. A manifest that references either tag fails to pull. Before you upgrade, update your [Kubernetes](../enterprise/platform-kubernetes) or [Docker Compose](../enterprise/platform-docker-compose) manifests: - Remove the `-unprivileged` or `-root` suffix from every frontend image reference. - Change the port. The image listens on `8000`, not `80`. In Kubernetes, set the container port and the frontend service `targetPort` to `8000`, and leave the service `port` at `80`. In Docker Compose, map the host port to container port `8000`. The templates you download in the [general upgrade steps](#general-upgrade-steps) already use these settings. See the [frontend image documentation](../enterprise/platform-kubernetes#seqera-frontend-unprivileged) for the security context, file system, and port differences. The [Helm chart](../enterprise/platform-helm) also requires this image. ## Studios container template version For 26.2, the default Studios container template version is **0.14**. The minimum supported version is **0.12**. If you customized your Studios container templates, update them to the 0.14 base images during the upgrade. Templates on a Connect version earlier than 0.12 are not supported. See the [Studios migration documentation](../studios/managing#migrate-a-studio-from-an-earlier-container-image-template). ## Data lineage available in all workspaces by default In 26.2, data lineage is available in every organization workspace by default. In 26.1, lineage was available only if you set `TOWER_LINEAGE_ALLOWED_WORKSPACES`. Availability does not change which runs generate lineage. Runs in a workspace generate lineage by default only after you configure the workspace lineage settings in **Settings > Workspace settings > Lineage** and turn on **Enable lineage by default**. The **Enable lineage** launch toggle overrides that setting for a single run. See [Enable data lineage](../data/data-lineage#enable-data-lineage). The [`TOWER_LINEAGE_ALLOWED_WORKSPACES`](./configuration/overview#data-features) environment variable controls lineage availability: | Value | Behavior | | --- | --- | | Unset (new default) | Lineage available in **all workspaces** | | `""` (empty string) | Lineage available in **all workspaces** | | Comma-separated workspace IDs | Lineage available only in the listed workspaces | To limit lineage to specific workspaces, set the variable to a comma-separated list of their IDs before you upgrade. ## Data lineage event ingestion moves from SQS to SNS Data lineage remains a preview feature and AWS-only. From 26.2, Platform no longer polls an Amazon Simple Queue Service (SQS) queue in your AWS account for lineage record notifications. Instead, the lineage bucket publishes object-created events to an Amazon Simple Notification Service (SNS) topic, which pushes them to a Platform webhook over HTTPS. No `sqs:*` grant remains in the documented permission set. If you plan to turn on lineage, grant the [lineage IAM permissions](../data/data-lineage#additional-iam-permissions-required) in addition to the existing [Seqera IAM permissions](../compute-envs/aws-batch#iam-user-creation). ### If lineage is already configured On the first startup after the upgrade, a one-off migration converts each lineage-enabled workspace that still uses the SQS transport. For an automatically provisioned workspace, the migration creates and configures the SNS topic, subscribes the Platform webhook, points the bucket notification rule to the topic, and then attempts to decommission the legacy SQS queue. While it runs, the migration needs the new SNS permissions, plus `sqs:DeleteQueue` on the legacy queue so that it can delete the queue. It needs no other SQS permission. After the migration completes, remove the SQS permissions from your IAM policies. Platform deletes the queue only after the workspace's SNS delivery is set up, so lineage events keep arriving through one channel or the other. Queue deletion is best effort. If Platform cannot delete the queue, for example because the credentials lack `sqs:DeleteQueue`, the only effect is a stale queue left in your AWS account. The queue no longer receives events, and Platform logs a warning with its URL so that you can delete it manually. A queue left behind does not count as a failed migration. The migration runs on the `cron` instance. After it processes the workspaces still on SQS, the `cron` log shows: ```console Lineage transport migration complete: migrated=X flagged=Y failed=Z ``` `migrated` counts automatically provisioned workspaces moved to SNS, `flagged` counts **Manual** workspaces left for you to reconfigure, and `failed` counts workspaces that could not be moved. When `failed` is `0`, every workspace has either moved to SNS or been flagged for you. A failed workspace keeps its queue and is retried on the next `cron` restart, or you can retry it from its lineage settings. See [If the migration reports an error](#if-the-migration-reports-an-error). If the installation is not reachable over public HTTPS, the migration logs an error, migrates nothing, and does not log the completion line. Set `TOWER_SERVER_URL` to a publicly resolvable HTTPS URL and restart. The migration is on by default. To turn it off, set `TOWER_LINEAGE_MIGRATE_SQS_TRANSPORT=false`. See [Configuration options](./configuration/overview#data-features). Platform cannot migrate customer-managed (**Manual**) workspaces, because it holds no permission over resources you own. The lineage settings of those workspaces show instructions to create an SNS topic, subscribe the Platform lineage webhook to it, and enter the topic ARN. :::note The migration does not modify the lineage records in your bucket, and no lineage is lost. The bucket is the source of truth for lineage records. In rare cases, such as a delay during setup, the lineage index can fall out of sync with the bucket. Platform can rebuild the index from the bucket. ::: ### If the migration reports an error If a workspace's credentials lack the required permissions, the migration leaves that workspace in an errored state and records the cause on its lineage settings page. Your bucket and the lineage data in it are not affected. To resolve: 1. Update the IAM role or user behind the workspace's lineage credentials to grant the [permissions listed on the data lineage page](../data/data-lineage#additional-iam-permissions-required). 1. Open **Settings > Workspace settings > Lineage** and select **Update** to retry provisioning. 1. If provisioning still does not complete, select **Disable lineage** and configure lineage again. This removes only the configuration and the automatically provisioned notification infrastructure. Platform re-indexes records already written to the bucket once event delivery is established. Retrying loses no lineage. ## General upgrade steps :::caution The database volume is persistent on the local machine by default if you use the `volumes` key in the `db` or `redis` section of your `docker-compose.yml` file to specify a local path to the DB or Redis instance. If your database is not persistent, back it up before you upgrade the application or the database. ::: 1. Back up the Seqera database. If you use the pipeline optimization service and its `groundswell` database is in a separate database instance, back up the `groundswell` database too. 1. Download the latest versions of your deployment templates and update your Seqera container versions: - [docker-compose.yml](./_templates/docker/docker-compose.yml) for Docker Compose deployments - [tower-cron.yml](./_templates/k8s/tower-cron.yml) and [tower-svc.yml](./_templates/k8s/tower-svc.yml) for Kubernetes deployments 1. **JVM memory defaults (recommended)**: The deployment templates you downloaded in the previous step include the following `JAVA_OPTS` environment variable to tune JVM memory settings: ```bash JAVA_OPTS="-Xms1000M -Xmx2000M -XX:MaxDirectMemorySize=800m -Dio.netty.maxDirectMemory=0 -Djdk.nio.maxCachedBufferSize=262144" ``` These baseline values suit most deployments with a moderate number of concurrent runs. :::tip You may need to tune these starting values for your workload. See [Backend memory requirements](./configuration/overview.mdx#backend-memory-requirements) for when and how to adjust them. ::: 1. If you use Studios, download and apply the latest versions of the Kubernetes manifests: - [proxy.yml](./_templates/k8s/data_studios/proxy.yml) - [server.yml](./_templates/k8s/data_studios/server.yml) :::warning If you customized the default Studios container template images, update them to the latest recommended versions. Seqera may not support templates that use a Connect version earlier than the one in the latest `proxy.yml` and `server.yml`. See the [Studios migration documentation](../studios/managing#migrate-a-studio-from-an-earlier-container-image-template) to migrate to the latest Connect server and client versions. ::: 1. Restart the application. 1. If you use a containerized database as part of your implementation: 1. Stop the application. 1. Upgrade the MySQL image. 1. Restart the application. 1. If you use Amazon RDS or another managed database service: 1. Stop the application. 1. Upgrade your database instance. 1. Restart the application. 1. If you use the pipeline optimization service and its `groundswell` database is separate from your Seqera database, update the MySQL image for the `groundswell` database instance while the application is down (during step 6 or 7). If both use the same database instance, the `groundswell` update happens automatically during the Seqera database update. ### Database migrations Database migrations run automatically during the upgrade. No manual steps are needed. ### Custom deployments - Run the `/migrate-db.sh` script in the `migrate-db` container to migrate the database schema. - Deploy Seqera with your usual procedure. ## Nextflow launcher image If you host your nf-launcher container image on a private image registry, copy the [nf-launcher image](https://quay.io/seqeralabs/nf-launcher:j21-26.04.x) to your private registry. Then set the launch container environment variable in your backend environment: ``` TOWER_LAUNCH_CONTAINER= ``` :::caution If you use AWS Batch, [configure a custom job definition](../enterprise/advanced-topics/custom-launch-container) and set `TOWER_LAUNCH_CONTAINER` to the job definition name instead. ::: --- ## Default version compatibility Seqera supports the two most recent major Seqera Platform versions (for example, 26.1.x and 26.2.x) at any given time. Each Seqera Platform version uses `nf-launcher` to set its baseline Nextflow version. To use a different Nextflow version in your pipeline runs, add a [pre-run script](../launch/advanced#pre-and-post-run-scripts) during launch. Seqera Platform may not work reliably with Nextflow versions other than the baseline. If you do not specify a Nextflow version in your configuration, Seqera Platform uses the baseline version listed in the following table: | Platform version | nf-launcher version | Nextflow version | Fusion version | Connect client version | | ---------------- | ------------------- | ---------------- | -------------- | ---------------------- | | 26.2.2 | j21-26.04 | 26.04 | 2.5 | 0.14.0 | | 26.2.1 | j21-26.04 | 26.04 | 2.5 | 0.14.0 | | 26.2.0 | j21-26.04 | 26.04 | 2.5 | 0.14.0 | | 26.1.5 | j21-26.04 | 26.04 | 2.5 | 0.12.0 | | 26.1.4 | j21-26.04 | 26.04 | 2.5 | 0.12.0 | | 26.1.3 | j21-26.04 | 26.04 | 2.5 | 0.12.0 | | 26.1.2 | j21-26.04 | 26.04 | 2.5 | 0.12.0 | | 26.1.0 | j21-26.04 | 26.04 | 2.5 | 0.12.0 | | 25.3.7 | j21-25.10.2 | 25.10.2 | 2.4 | 0.11.0 | | 25.3.6 | j21-25.10.2 | 25.10.2 | 2.4 | 0.11.0 | | 25.3.4 | j21-25.10.2 | 25.10.2 | 2.4 | 0.9.0 | | 25.3.1 | j21-25.10.2 | 25.10.2 | 2.4 | 0.9.0 | | 25.3.0 | j21-25.04.8 | 25.04.8 | 2.4 | | | 25.2.4 | j21-25.04.3 | 25.04.3 | 2.4 | | | 25.2.3 | j21-25.04.3 | 25.04.3 | 2.4 | | | 25.2.2 | j21-25.04.3 | 25.04.3 | 2.4 | | | 25.2.1 | j21-25.04.3 | 25.04.3 | 2.4 | | | 25.2.0 | j21-25.04.3 | 25.04.3 | 2.4 | | | 25.1.5 | j17-24.10.9-b1 | 24.10.9 | 2.4 | | | 25.1.4 | j17-24.10.9-b1 | 24.10.9 | 2.4 | | | 25.1.3 | j17-24.10.9-b1 | 24.10.9 | 2.4 | | | 25.1.1 | j17-24.10.5 | 24.10.5 | 2.4 | | | 25.1.0 | j17-24.10.5 | 24.10.5 | 2.4 | | | 24.2.7 | j17-24.10.9-a1 | 24.10.9 | 2.4 | | | 24.2.6 | j17-24.10.9-a1 | 24.10.9 | 2.4 | | | 24.2.4 | j17-24.10.4 | 24.10.4 | 2.4 | | | 24.2.3 | j17-24.10.4 | 24.10.4 | 2.4 | | | 24.2.2 | j17-24.10.2 | 24.10.2 | 2.4 | | | 24.2.1 | j17-24.10.2 | 24.10.2 | 2.4 | | | 24.2.0 | j17-24.10.2 | 24.10.2 | 2.4 | | `nf-launcher` versions prefixed with `j21` use Java 21, and versions prefixed with `j17` use Java 17. --- ## Deploy Platform Deploy Seqera Platform Enterprise in your organization's own infrastructure, then access it through the UI, the API, the CLI, or Nextflow. [Seqera Platform Enterprise](../enterprise/overview) is installed in an organization's own cloud or on-premises infrastructure. It includes: - Monitoring, logging, and observability - Pipeline execution Launchpad - Cloud resource provisioning - Pipeline actions and event-based execution - LDAP and OpenID authentication - Enterprise role-based access control (RBAC) - Full-featured API - Dedicated support for Nextflow and Seqera Platform To install Platform in your organization's infrastructure, [contact us](https://cloud.seqera.io/demo/) for a demo to discuss your requirements. ## Access Platform You can access your Seqera instance through the UI, the [API](https://docs.seqera.io/platform-api), the [CLI](https://docs.seqera.io/platform-cli), or in Nextflow directly using the `-with-tower` option. ### Web-based UI To get started in the Platform UI: 1. Sign in to your Seqera Platform instance at the URL provided by your administrator. :::note Login sessions remain active while the browser window is open and active. After you close the browser window, Platform signs you out within 6 hours by default. ::: 2. Create and configure a new [compute environment](../compute-envs/overview). 3. Start [launching pipelines](../launch/launchpad). ### Seqera API See [API](https://docs.seqera.io/platform-api). :::tip To find the organization and workspace IDs that API endpoints use, see [Automation](./quickstart-demo/automation#find-your-organization-and-workspace-ids). ::: ### Seqera CLI See [CLI](https://docs.seqera.io/platform-cli). ### Nextflow `-with-tower` If you run Nextflow directly in an existing environment, add the `-with-tower` option to your run command to use Platform capabilities: 1. Sign in to your Seqera Platform instance. 2. In your personal workspace, go to the user menu and select **Settings** > **Your tokens**. 3. Select **Add token**. 4. Enter a unique name for your token, then select **Add**. 5. Copy and store your token securely. :::caution Platform displays the access token only once. Save the token value before you close the **Personal Access Token** window. ::: 6. Open a terminal window and create environment variables to store the Seqera access token and Nextflow version. Replace `` with your new token. ```bash export TOWER_ACCESS_TOKEN= export NXF_VER=26.04.6 ``` :::note Bearer token support requires Nextflow version 20.10.0 or later. Set the version with the `NXF_VER` environment variable. ::: 7. To submit a pipeline to a [workspace](../orgs-and-teams/workspace-management) using Nextflow, add the workspace ID to your environment: ```bash export TOWER_WORKSPACE_ID=000000000000000 ``` To find your workspace ID, select your organization in Seqera and navigate to the **Workspaces** tab. 8. Run your Nextflow pipeline with the `-with-tower` flag: ```bash nextflow run main.nf -with-tower ``` Replace `main.nf` with the filename of your Nextflow script. You can now monitor your workflow runs in the Seqera interface. To configure and execute Nextflow pipelines in cloud environments, see [compute environments](../compute-envs/overview). :::tip See the [Nextflow documentation](https://docs.seqera.io/nextflow/config.html?highlight=tower#scope-tower) for further run configuration options using Nextflow configuration files. ::: --- ## Run a pipeline On this page, learn how to run a pipeline with sample data and get started running your own pipelines. :::tip [**Sign up**](https://tower.nf "Seqera Platform") to try Seqera for free, or request a [**demo**](https://cloud.tower.nf/demo/ "Seqera Platform Demo") for deployments in your own on-premises or cloud environment. ::: The Community Showcase [Launchpad](../launch/launchpad) is an example workspace provided by Seqera. The showcase is pre-configured with compute environments, credentials, and pipelines to start running Nextflow workflows immediately. A pipeline consists of a pre-configured workflow repository, compute environment (with 100 free CPU hours), and launch parameters. From version 23.1.3, the Community Showcase comes pre-loaded with two AWS Batch compute environments to run showcase pipelines. ### Components - [Datasets](../data/datasets) are collections of versioned, structured data (usually in the form of a samplesheet) in CSV or TSV format. A dataset is used as the input for a pipeline run. Sample datasets are used in pipelines with the same name, e.g., the _nf-core-rnaseq-test_ dataset is used as input when you run the _nf-core-rnaseq_ pipeline. - [Compute environments](../compute-envs/overview) are the platforms where workflows are executed. A compute environment consists of access credentials, configuration settings, and storage options for the environment. - [Credentials](../credentials/overview) are the authentication keys Seqera uses to access compute environments, private code repositories, and external services. Credentials are SHA-256 encrypted before secure storage. The Community Showcase includes all the credentials you need to run pipelines in the included AWS Batch compute environments. - [Secrets](../secrets/overview) are retrieved and used during pipeline execution. In your private or organization workspace, you can store the access keys, licenses, or passwords required for your pipeline execution to interact with third-party services. The secrets included in the Community Showcase contain license keys to run _nf-dragen_ and _nf-sentieon_ pipelines in the showcase compute environments. ## Run a pipeline with sample data 1. From the [Launchpad](../launch/launchpad), select a pipeline to view the pipeline detail page. _nf-core-rnaseq_ is a good first pipeline example. 2. Optional: Select the URL under **Workflow repository** to view the pipeline code repository in another tab. 3. Select **Launch** from the pipeline detail page. 4. On the **Launch pipeline** page, enter a unique **Workflow run name** or use the pre-filled random name. 5. Optional: Enter labels to be assigned to the run in the **Labels** field. 6. Under **Input/output options**, select the dataset named after your chosen pipeline from the drop-down under **input**. 7. Under **outdir**, specify an output directory where run results will be saved. This must be an absolute path to storage on cloud infrastructure and defaults to `./results`. 8. Under **email**, enter an email address where you wish to receive the run completion summary. 9. Under **multiqc_title**, enter a title for the MultiQC report. This is used as both the report page header and filename. The remaining launch form fields will vary depending on the pipeline you have selected. Parameters required for the pipeline to run are pre-filled by default, and empty fields are optional. Once you've filled the necessary launch form details, select **Launch**. Your new run will be displayed at the top of the list in the **Runs** tab with a **submitted** status. Select the run name to navigate to the run detail page and view the configuration, parameters, status of individual tasks, and run report. ## Run your own pipelines To run pipelines on your own infrastructure, you first need to create your own organization. * [Organizations](../orgs-and-teams/organizations) are the top-level structure in Seqera. They contain the building blocks of your organizational infrastructure. * [Workspaces](../orgs-and-teams/workspace-management) are where resources are managed. All team members can access the organization workspace. In addition to this, each user has a unique personal workspace to manage resources such as pipelines, compute environments, and credentials. * [Teams](../orgs-and-teams/organizations) are collections of members. * [Members](../administration/overview#members) belong to an organization and can have different levels of access across workspaces. You can create multiple workspaces within an organization context and associate each of these workspaces with dedicated teams of users, while providing fine-grained access control for each of the teams. See [Administration](../orgs-and-teams/organizations). --- ## Production checklist This guide is for Platform administrators preparing a Seqera deployment for production use by scientific teams. It covers common configuration decisions, policies, and checks to consider before you run production workloads. Because every deployment is different, your Seqera account team can help you tailor these recommendations to your environment and workloads. Cloud infrastructure setup, such as networking, IAM, and compute provisioning, is the responsibility of your infrastructure team and is not covered here. ## Organizations and workspaces Organizations are the top-level structure in Seqera Platform and contain workspaces, members, and teams. You can create multiple organizations, each with multiple workspaces, to customize resource use and maintain access control across teams. Best practices for organizations and workspaces include: - Plan your organization and workspace structure, considering the roles and work streams you expect to start with and scale to. - Use separate workspaces to isolate production from development and test environments. See [Organizations](../orgs-and-teams/organizations) for more information. ## Users and roles Roles define an organization member's access and permissions within Platform. Each member has an organization role and can operate in one or more workspaces, where a participant role governs what they can do within that workspace. Best practices for users and roles include: - Map out expected users and their roles before go-live to ensure your access model is scalable. - Assign roles at the level of access each user actually requires rather than granting broad permissions by default. - Limit organization owner assignment to users responsible for managing members, teams, and organization-level settings. See [User roles](../orgs-and-teams/roles) for more information. ## Version pinning and compatibility Incompatibilities between Nextflow and Platform versions are a leading cause of production failures, and often only surface during pipeline resumption after an interruption. Best practices for version pinning and compatibility include: - Set the Nextflow version in the compute environment configuration before handing workspaces to scientific teams, and document it alongside the Seqera Platform version in use. - Avoid performing Nextflow and Platform upgrades simultaneously, and validate all version changes in a non-production environment first. - Coordinate with pipeline developers to confirm the current working version combination and agree on a rollback plan before you schedule any upgrades. - Before promoting a version change to production, ask pipeline teams to test resumption explicitly: launch a representative pipeline, interrupt it, and confirm it resumes from the last successful task. :::warning Resumption failure after an upgrade typically indicates a version incompatibility issue. If pipeline teams cannot resume after a version upgrade, roll back to the last documented working version combination before investigating further. ::: ## Cache management Nextflow uses content-addressed caching to resume pipelines from the last successful task. Cache misconfiguration can lead to unexpected behavior during pipeline resumption. The administrator's role is to configure the storage environment that cache depends on, and to coordinate with pipeline teams to validate cache behavior before go-live. Best practices for cache management include: - Configure storage lifecycle policies to ensure that intermediate work directory objects are not removed while pipelines are still running or may need to resume. Coordinate these policies with your cloud team before go-live. - Ask pipeline teams to validate cache integrity before go-live: run a representative pipeline, run it again, and confirm that cache hits occur and outputs match a clean run. - Schedule Platform or Nextflow upgrades with pipeline teams in advance, and plan for the first post-upgrade production runs to take longer and cost more if cache is invalidated by a hash algorithm change. Your Seqera account team can help assess cache configuration for your specific workloads and storage setup. :::warning After a Nextflow or Platform version upgrade, hash algorithms or cache key generation may change, causing all previously cached tasks to re-run. Inform pipeline teams before any upgrade so they can plan capacity and schedule accordingly; this is an expected behavior, but it may cause alarm if it happens unexpectedly. ::: ## Credentials and token lifecycle Credentials in Seqera Platform require active management. Expired or rotated credentials that are not updated in Platform are a common cause of silent pipeline failures. Before you go live: - Identify all credentials used by production pipelines: cloud provider credentials, Git tokens, container registry credentials, and API tokens. - Record when each credential was created and when it expires. - Assign a named owner responsible for rotating each credential. **When rotating credentials** 1. Add the new credential to the correct Seqera organization and workspaces. 2. Launch a test pipeline using the new credential and confirm it runs successfully. 3. Remove or deactivate the old credential only after step 2 is confirmed. :::warning Do not rotate credentials during active pipeline runs. Schedule rotations during maintenance windows. ::: Use [Pipeline Secrets](../secrets/overview) to manage sensitive values such as API keys for third-party services. Secrets are injected at runtime and are not exposed in pipeline logs or configuration files. ## Compute environment permissions Permissions within shared compute environments can cause unexpected behavior, particularly when multiple teams use the same workspace. Best practices for compute environment permissions include: - Use dedicated compute environments for production and avoid sharing production compute environments with development or test workloads. - Assign workspace roles at the level of access each user actually requires. The **Launch** role is appropriate for most researchers running established pipelines; **Maintain** is for users who need to configure compute environments and pipelines. - Use separate workspaces if your organization requires run isolation between teams or projects. Users in the same workspace can see and cancel each other's pipeline runs. :::note Admin-level workspace access grants the ability to modify compute environments and credentials, which can affect all pipelines in the workspace. Assign Admin only to users who are responsible for workspace configuration. ::: ## Compute environment sizing Correctly sizing compute environments before go-live prevents resource contention, job failures, and unexpected costs in production. Best practices for compute environment sizing include: - A typical starting range for max CPUs is 2000 to 5000, depending on your workload volume and concurrency needs. Your Seqera account team can advise on the best sizing for your environment. - Consider enabling Fusion v2 with fast instance storage (NVMe) for I/O-intensive workloads on AWS. In tested benchmarks, this showed a 34% reduction in total pipeline runtime and up to 49% reduction in CPU hours compared to plain S3 storage. See [RNA-Seq performance benchmarks](../getting-started/rnaseq#nf-corernaseq-performance-in-platform) for details. - Use the [pipeline optimization feature](../pipeline-optimization/overview) to right-size resource allocations based on actual usage data. After a successful run, select the lightbulb icon next to the pipeline in the Launchpad to view and apply an optimized configuration profile. - For GPU workloads such as protein structure prediction, use GPU-enabled instance families (`g4dn`, `g5`, or `p3` on AWS) and ensure the GPU ECS AMI is enabled in the compute environment configuration. :::warning Studios do not support AWS Fargate. If you share a compute environment between pipelines and Studios, do not enable **Use Fargate for head job**. Enabling Fargate on a shared compute environment will prevent Studios sessions from starting. ::: :::note Studio sessions compete for compute resources with pipeline runs in the same compute environment. For production workloads, use a dedicated compute environment for pipelines, or ensure the shared environment has sufficient capacity to accommodate both concurrently. ::: See [RNA-Seq](../getting-started/rnaseq) and [Protein structure prediction](../getting-started/proteinfold) for workload-specific compute environment recommendations. ## Cost tagging Without resource labels, cloud billing reports cannot attribute compute costs to specific teams, projects, or pipelines. Best practices for cost tagging include: - Define a tagging strategy before running production workloads. At minimum, tag by `environment`, `team`, and `pipeline`. Add `project` or `cost_center` tags if you need chargeback reporting. - Use [dynamic resource labels](../resource-labels/overview) to apply pipeline-specific tags to AWS Batch jobs automatically at run time. This enables cost attribution at the individual run level without manual configuration. :::warning Cancelling a pipeline run in Seqera Platform does not guarantee immediate termination of the underlying cloud compute jobs. Configure spend alerts in your cloud provider's billing tools independently of Platform, so that runaway compute costs are detected even if Platform does not surface them. ::: :::note The cost estimator in Seqera Platform is for estimation purposes and doesn't account for inefficiencies in how cloud batch executors provision instances. For billing, budgeting, or chargeback, use your cloud provider's native cost reporting tools: AWS Cost Explorer, GCP Billing, or Azure Cost Management. ::: ## Cost management and alerts Managing compute spend before your workloads go live reduces the risk of unexpected charges in production. Best practices for cost management and alerts include: - Enable billing exports to your cloud provider's analytics tooling before running production workloads: AWS Cost Explorer or S3 export, GCP billing export to BigQuery, or Azure Cost Management. These give you the raw data needed to investigate unexpected charges. - Set budget alerts in your cloud provider's billing tools to detect unexpected daily or weekly spend changes. On AWS, configure CloudWatch billing alarms; on GCP, use Cloud Monitoring budget alerts; on Azure, use Cost Management alert rules. - On AWS, use [dynamic resource labels](../resource-labels/overview) to tag Batch jobs with pipeline-specific values at run time. Dynamic labels do not appear in AWS Cost Explorer's graphical UI. Costs are tracked via AWS split cost allocation data in Cost and Usage Reports (CUR). Enable split cost allocation in the AWS billing console before expecting to see per-pipeline costs. See the [Seqera blog post on AWS labels cost tracking](https://seqera.io/blog/aws-labels-cost-tracking/) for setup guidance. - On GCP, apply labels to Cloud Storage buckets, Filestore instances, and Compute Engine VMs in addition to Cloud Batch jobs. Label all resources consistently so BigQuery billing exports can attribute costs at the workload level. - Account for CloudWatch fees separately on AWS as these are not included in Seqera Platform run cost estimates. ## Spot instance retry strategy Spot reclamation interrupts pipeline execution when cloud providers reclaim capacity. The administrator configures the compute environment and Platform settings that make Spot workloads resilient; pipeline developers are responsible for the Nextflow-level retry configuration in their pipelines. Both should be in place before production runs begin. **Administrator configuration** - Decide on the Spot vs On-Demand provisioning model for each production compute environment and configure the retry logic accordingly. - Enable [Fusion Snapshots](https://docs.seqera.io/fusion/guide/snapshots) in the compute environment for Spot workloads. When a Spot instance is reclaimed, Fusion Snapshots allows the interrupted task to resume automatically from a checkpoint rather than restart from scratch. **Pipeline developer configuration** - `aws.batch.maxSpotAttempts` controls how many times AWS Batch retries a task before Nextflow sees it as a failure. See [Handle retries in AWS](../compute-envs/aws-spot-interruptions#handle-retries-in-aws-by-setting-awsbatchmaxspotattempts). - `errorStrategy` and `maxRetries` in Nextflow handle failures that survive all AWS-native retries. See [Handle retries in Nextflow](../compute-envs/aws-spot-interruptions#handle-retries-in-nextflow-by-setting-errorstrategy-and-maxretries). - Implement Spot to On-Demand fallback logic for tasks where interruption is unacceptable. See [Spot to On-Demand fallback](../compute-envs/aws-spot-interruptions#implement-spot-to-on-demand-fallback-logic). :::note `aws.batch.maxSpotAttempts` and Nextflow's `maxRetries` operate at independent layers. AWS-native Spot retries happen silently — Nextflow only sees the task as failed after all AWS retries are exhausted. Pipeline developers must set both values intentionally; neither is configured by Platform automatically. ::: :::note On GCP, Spot preemption notices can be as short as 6 seconds (compared to 120 seconds on AWS), making Fusion Snapshots especially important. Add the following to your Nextflow configuration: ```groovy fusion { enabled = true snapshots = true } ``` ::: ## Connectivity requirements Seqera Platform requires outbound connectivity from compute worker nodes to specific endpoints. Blocked connections cause pipelines to fail to start or stall mid-run. Worker nodes must be able to reach your cloud provider's service endpoints (Batch, EC2, or Compute Engine, object storage, container registry), any container registries used by your pipelines, and any external data sources accessed at runtime. In addition, the following Seqera-operated domains must be reachable from your worker nodes. If your firewall supports DNS wildcards, add `*.seqera.io.cdn.cloudflare.net`. Otherwise, add each domain individually: - `wave.seqera.io` and `community.wave.seqera.io` - `fusionfs.seqera.io` - `nf-xpack.seqera.io` - `cr.seqera.io`, `public.cr.seqera.io`, and `auth.cr.seqera.io` - `hub.seqera.io` - `licenses.seqera.io` - `registry.nextflow.io` — required from Nextflow 25.10 onwards - `api.multiqc.info` - For Seqera Cloud deployments: `cloud.seqera.io` and `api.cloud.seqera.io`. - For Enterprise (self-hosted) deployments: your Platform instance hostname. If your allowlist is based on IP addresses rather than DNS names, allow all Cloudflare IP ranges listed at [https://www.cloudflare.com/ips/](https://www.cloudflare.com/ips/). For a dynamic list of Seqera Platform egress IPs, query `https://meta.seqera.io`. If your environment uses SSL inspection or a corporate proxy, verify that it does not interfere with connections to Seqera Platform endpoints, object storage, or container registries. For Enterprise deployments in restricted or air-gapped environments, configure proxy settings on both the Platform instance and worker nodes, and provision an internal plugin registry to replace `registry.nextflow.io`. :::warning From Nextflow 25.10, `registry.nextflow.io` is required for plugin resolution and is not included in older firewall allowlists. Pipelines will fail to start after upgrading to 25.10 if this domain is blocked. If your organization requires an internal plugin registry instead, see the [Nextflow 25.10 migration guide](https://www.nextflow.io/docs/latest/migrations/25-10.html). ::: :::note If pipelines fail to start or tasks cannot pull container images, connectivity to one of the above endpoints is the most common cause. Check worker node outbound access before investigating other causes. See [Firewall configuration](../enterprise/advanced-topics/firewall-configuration) for the full allowlist reference. ::: --- ## Protein structure prediction This guide details how to perform best-practice analysis for protein 3D structure prediction on an AWS Batch compute environment in Platform. It includes: - Creating AWS Batch compute environments to run your pipeline and downstream analysis - Adding the *nf-core/proteinfold* pipeline to your workspace - Importing your pipeline input data - Launching the pipeline and monitoring execution from your workspace - Setting up a custom analysis environment with Studios :::info[**Prerequisites**] You will need the following to get started: - [Admin](../orgs-and-teams/roles) permissions in an existing organization workspace. See [Set up your workspace](./workspace-setup) to create an organization and workspace from scratch. - An existing AWS cloud account with access to the AWS Batch service. - Existing access credentials with permissions to create and manage resources in your AWS account. See [IAM](../compute-envs/aws-batch#iam-user-creation) for guidance to set up IAM permissions for Platform. ::: ## Compute environment The compute and storage requirements for protein structure prediction depend on the number and length of protein sequences being analyzed and the size of the database used for prediction by the deep learning models, such as AlphaFold2 and ColabFold. Input sequences typically range from a few kilobytes for single proteins to several megabytes for large datasets, and reference databases can be extremely large, from 100 GB to several TB. Protein folding pipelines generate intermediate files during execution, such as for alignments and feature representations, the sizes of which vary based on the number of sequences and the complexity of the analysis. Given the data sizes and computational intensity, production pipelines perform best with NVIDIA A10 or larger GPUs and low-latency, high-throughput cloud storage file handling. ### GPUs The *nf-core/proteinfold* pipeline performs protein folding prediction using one of three deep learning models: AlphaFold2, ColabFold, or ESMFold. The computationally intensive tasks for protein structure prediction perform better on GPUs due to their ability to handle large matrix operations efficiently and perform parallel computations. GPUs can dramatically reduce the time required for protein structure predictions, making it feasible to analyze larger datasets or perform more complex simulations. Platform supports the allocation of both CPUs and GPUs in the same compute environment. For example, specify `m6id`, `c6id`, `r6id`, `g5`, `p3` instance families in the **Instance types** field when creating your AWS Batch compute environment. See [Create compute environment](#create-compute-environment) below. When you launch *nf-core/proteinfold* in Platform, enable **use_gpu** to instruct Nextflow to run GPU-compatible pipeline processes on GPU instances. See [Launch pipeline](#launch-pipeline) below. ### Fusion file system The [Fusion](../supported_software/fusion/overview) file system enables seamless read and write operations to cloud object stores, leading to simpler pipeline logic and faster, more efficient execution. While Fusion is not required to run nf-core/proteinfold, it significantly enhances I/O-intensive tasks and eliminates the need for intermediate data copies, which is particularly beneficial when working with the large databases used by deep learning models for prediction. Fusion works best with AWS NVMe instances (fast instance storage) as this delivers the fastest performance when compared to environments using only AWS EBS (Elastic Block Store). Batch Forge selects instances automatically based on your compute environment configuration, but you can optionally specify instance types. To enable fast instance storage, you must select EC2 instances with NVMe SSD storage (`g4dn`, `g5`, or `P3` families or greater). :::note Fusion requires a license for use in Seqera Platform compute environments or directly in Nextflow. See [Fusion licensing](https://docs.seqera.io/fusion/licensing) for more information. ::: ### Create compute environment :::info The same compute environment can be used for pipeline execution and running your Studios notebook environment, but Studios does not support AWS Fargate. To use this compute environment for both *nf-core/proteinfold* execution and your Studio, leave **Enable Fargate for head job** disabled and include a CPU-based EC2 instance family (`c6id`, `r6id`, etc.) in your **Instance types**. Alternatively, create a second basic AWS Batch compute environment and a Studio with at least 2 CPUs and 8192 MB of RAM. ::: From the **Compute Environments** tab in your organization workspace, select **Add compute environment** and complete the following fields: | **Field** | **Description** | |---------------------------------------|------------------------------------------------------------| | **Name** | A unique name for the compute environment. | | **Platform** | AWS Batch | | **Credentials** | Select existing credentials, or **+** to create new credentials.| | **Access Key** | AWS access key ID. | | **Secret Key** | AWS secret access key. | | **Region** | The target execution region. | | **Pipeline work directory** | An S3 bucket path in the same execution region. | | **Enable Wave Containers** | Use the Wave containers service to provision containers. | | **Enable Fusion v2** | Access your S3-hosted data via the Fusion v2 file system. | | **Enable fast instance storage** | Use NVMe instance storage to speed up I/O and disk access. Requires Fusion v2.| | **Config Mode** | Batch Forge | | **Provisioning Model** | Choose between Spot and On-demand instances. | | **Max CPUs** | Sensible values for production use range between 2000 and 5000.| | **Enable Fargate for head job** | Run the Nextflow head job using the Fargate container service to speed up pipeline launch. Requires Fusion v2. Do not enable for Studios compute environments. | | **Use Amazon-recommended GPU-optimized ECS AMI** | When enabled, Batch Forge specifies the most current AWS-recommended GPU-optimized ECS AMI as the EC2 fleet AMI when creating the compute environment. | | **Allowed S3 buckets** | Additional S3 buckets or paths to be granted read-write permission for this compute environment. For the purposes of this guide, add `s3://proteinfold-dataset` to grant compute environment access to the DB and params used for prediction by AlphaFold2 and ColabFold. | | **Instance types** | Specify the instance types to be used for computation. You must include GPU-enabled instance types (`g4dn`, `g5`) when the Amazon-recommended GPU-optimized ECS AMI is in use. Include CPU-based instance families for Studios compute environments. | | **Resource labels** | `name=value` pairs to tag the AWS resources created by this compute environment.| ## Add pipeline to Platform :::info The [*nf-core/proteinfold*](https://github.com/nf-core/proteinfold) pipeline is a bioinformatics best-practice analysis pipeline for Protein 3D structure prediction. ![nf-core/proteinfold subway map](./_images/nf-core-proteinfold_metro_map_1.1.0.png) ::: [Seqera Pipelines](https://seqera.io/pipelines) is a curated collection of quality open source pipelines that can be imported directly to your workspace Launchpad in Platform. Each pipeline includes a curated test dataset to use in a test run to confirm compute environment compatibility in just a few steps. To use Seqera Pipelines to import the *nf-core/proteinfold* pipeline to your workspace: