Skip to main content

Kubernetes

The k8s executor runs pipeline tasks on a Kubernetes cluster. It submits each task as a Kubernetes job, or as a Kubernetes pod when k8s.computeResourceType is set to Pod.

To enable this executor, set process.executor = 'k8s' in the nextflow.config file. You can launch the pipeline from a pod in the cluster, or from any machine that can connect to the cluster when using Fusion. Every pipeline process must specify a container image using the container directive. The pipeline work directory must be in a shared persistent volume, or in object storage when using Fusion.

Use the following process directives to control resource requests and other job characteristics:

See the k8s config scope for the full list of configuration options.

Overview​

The main abstraction in Kubernetes is the pod, which defines the desired state of one or more containers, such as the required compute resources, storage, and network configuration.

Kubernetes also abstracts storage provisioning through persistent volumes. Containers use persistent volumes to access the underlying storage systems in a portable way.

When using the k8s executor, Nextflow submits each task as a Kubernetes job (or pod, depending on k8s.computeResourceType). Nextflow itself typically runs in a pod in the same cluster, which mounts the same persistent volume as the tasks (see Running in a pod). When using Fusion, Nextflow can run anywhere that can connect to the cluster (see Launch with Fusion).

Nextflow k8s

Getting started​

  1. Create a Persistent Volume with ReadWriteMany access mode in the Kubernetes cluster, and a Persistent Volume Claim to access it. Nextflow uses this volume for the work directory and pipeline outputs.

    tip

    When you use Wave and Fusion, you don't need a shared file system or persistent volume claim. See Launch with Fusion for more information.

  2. Make sure Nextflow can connect to the Kubernetes cluster. Nextflow uses the Kubernetes configuration file at $HOME/.kube/config or the file specified by the KUBECONFIG environment variable. You can verify the configuration with the following command:

    $ kubectl cluster-info
    Kubernetes master is running at https://your-host:6443
    KubeDNS is running at https://your-host:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

  3. Make sure that every process in your pipeline specifies a container with the container directive.

  4. Add the following settings to your Nextflow configuration:

    • Set process.executor to k8s.
    • Set k8s.storageClaimName to the name of the persistent volume claim.
    • Set k8s.storageMountPath to the path where the volume is mounted (default: /workspace).
    • Set workDir to a path in the mounted volume, for example, /workspace/work.

  5. Launch your pipeline from a pod in the cluster that mounts the same persistent volume claim. See Running in a pod for details.

For example:

process {
    executor = 'k8s'
    container = 'quay.io/biocontainers/salmon'
}

workDir = '/workspace/work'

k8s {
    storageClaimName = '<pvc-name>'
    storageMountPath = '/workspace'
}

Running in a pod​

You can run Nextflow directly from a pod running in a Kubernetes cluster. In this case, use the plain Nextflow run command and specify the k8s executor and the required persistent volume claim in the nextflow.config file:

process {
    executor = 'k8s'
}

k8s {
    storageClaimName = 'vol-claim'
    storageMountPath = '/mount/path'
    storageSubPath = '/my-data'
}

In the above snippet, replace vol-claim with the name of an existing persistent volume claim, /mount/path with the mount path (default: /workspace), and storageSubPath with the directory in the volume to mount (default: /).

warning

The running pod must have been created with the same persistent volume claim name and mount as the one specified in your Nextflow configuration file. The run command does not support the -v option.

tip

You can also mount multiple volumes using the k8s.pod config option. For example:

k8s.pod = [ [volumeClaim: "other-pvc", mountPath: "/other" ]]

Launch with Fusion​

With Fusion, you can deploy a Nextflow pipeline to a remote (or local) Kubernetes cluster without a shared file system or persistent volume claim.

See Amazon EKS for a guide to set up Fusion in Amazon EKS compute environments and Google Kubernetes Engine (GKE) for a guide to set up Fusion in GKE compute environments.

Launch with kuberun​

The kuberun command deploys the Nextflow runtime as a pod and runs a pipeline from it. It is obsolete and no longer maintained. Use Running in a pod or Launch with Fusion instead.

On this Page