Skip to main content
Version: 22.4

Data Privacy

Your data

Your data stays strictly within your infrastructure itself. When you launch a workflow through Tower, you need to connect your infrastructure (HPC/VMs/K8s) by creating the appropriate credentials and compute environment in a workspace.

Tower then uses this configuration to trigger a Nextflow workflow within your infrastructure similar to what is done via the Nextflow CLI, therefore Tower does not manipulate any data itself and no data is transferred to the infrastructure where Tower is running.

It may be possible to access some data within your storage from the Nextflow Tower interface - for example, viewing logs and reports generated in a pipeline run - however, this data is never stored within the Tower infrastructure.

Metadata stored by Nextflow Tower

Workflow execution metadata is sent by the Nextflow runtime to Nextflow Tower when:

  • Launching workflow with Tower
  • Using the -with-tower option at the command line
  • When a Nextflow Tower is specified in the Nextflow config

The following sections describe the data structure and metadata fields collected by Tower.

Workflow metadata

The following metadata fields are collected and stored by the Tower backend during a workflow execution:

NameDescription
command_lineThe command line used to launch the workflow execution
commit_idThe workflow project commit Id at the time of the execution
completeThe workflow execution completion timestamp
config_filesThe nextflow config file paths(s) involved in the workflow execution
config_textThe nextflow config content used for the workflow execution. Note: secrets, such as, AWS keys are stripped and not included in this field.
containerThe container image name(s) used for the pipeline execution
container_engineThe container engine name used for the pipeline execution
durationThe workflow execution overall duration (wall time)
error_messageThe error message reported in the case of nextflow execution failure
error_reportThe extended error message reported in case of workflow execution error.
exit_statusThe workflow execution (POSIX) exit code
home_dirThe launching user home directory path
launch_dirThe workflow launching directory path
manifest_authorThe workflow project author as defined in the nextflow config manifest file
manifest_default_branchThe workflow project default Git branch as defined in the nextflow config manifest file
manifest_descriptionThe workflow project description as defined in the nextflow config manifest file
manifest_gitmodulesThe workflow project Git submodule flag in the nextflow config manifest file
manifest_home_pageThe workflow project Git home page as defined in the nextflow config manifest file
manifest_main_scriptThe workflow project main script file name as defined in the nextflow config manifest file
manifest_nameThe workflow project name as defined in the nextflow config manifest file
manifest_nextflow_versionThe workflow project required Nextflow version defined in the nextflow config manifest file
manifest_versionThe workflow project version string as defined in the nextflow config manifest file
nextflow_buildThe build number of the Nextflow runtime used to launch the workflow execution
nextflow_timestampThe build timestamp of the Nextflow runtime used to launch the workflow execution
nextflow_versionThe version string of the Nextflow runtime used to launch the workflow execution
paramsThe workflow params used to launch the pipeline execution
profileThe workflow config profile string used for the pipeline execution
project_dirThe directory path where the workflow scripts are stored
project_nameThe workflow project name
repositoryThe workflow project repository
resumeThe flag set when a resume execution was submitted
revisionThe workflow project revision number
run_nameThe workflow run name as given by the Nextflow runtime
script_fileThe workflow script file path
script_idThe workflow script checksum number
script_nameThe workflow script filename
session_idThe workflow execution unique UUID as assigned by the Nextflow runtime
startThe workflow execution start timestamp
stats_cached_countThe number of cached tasks upon completion
stats_cached_durationThe aggregate time of cached tasks upon completion
stats_cached_pctThe percentage of cached tasks upon completion
stats_compute_time_fmtThe overall compute time as a formatted string
stats_failed_countThe number of failed tasks upon completion
stats_failed_count_fmtThe number of failed tasks upon completion as a formatted string
stats_failed_durationThe aggregate time of failed tasks upon completion
stats_failed_pctThe percentage of failed tasks upon completion
stats_ignored_countThe number of ignored tasks upon completion
stats_ignored_count_fmtThe number of ignored tasks upon completion as a formatted string
stats_ignored_pctThe percentage of ignored tasks upon completion
stats_succeed_countThe number of succeeded tasks upon completion
stats_succeed_count_fmtThe number of succeeded tasks upon completion as a formatted string
stats_succeed_durationThe aggregate time of succeeded tasks upon completion
stats_succeed_pctThe percentage of succeeded tasks upon completion
statusThe workflow execution status
submitThe workflow execution submission timestamp
successThe flag reporting whether the execution completed successfully
user_nameThe POSIX user name launching that launched the workflow execution
work_dirThe workflow execution scratch directory path

Task Metadata

NameDescription
attemptNumber of Nextflow execution attempts of the task
cloud_zoneCloud zone where the task execution was allocated
completeTask execution completion timestamp
containerContainer image name used to execute the task
costEstimated task compute cost
cpusNumber of CPUs requested
diskAmount of disk storage requested
durationAmount of time for the task completion
envTask execution environment variables
error_actionAction applied on task failure
executorExecutor requested for the task execution
exit_statusTask POSIX exit code on completion
hashTask unique hash code
inv_ctxtNumber of involuntary context switches
machine_typeCloud virtual machine type
memoryAmount of memory requested
moduleEnvironment Module requested
nameTask unique name
native_idTask unique ID as assigned by the underlying execution platform
pcpuPercentage of CPU used to compute the task
peak_rssPeak of real memory during the task execution
peak_vmemPeak of virtual memory during the task execution
pmemPercentage of memory used to compute the task
price_modelThe cloud price model applied for the task
processThe nextflow process name
queueThe compute queue name requested
rcharNumber of bytes the process read, using any read-like system call from files, pipes, tty, etc.
read_bytesNumber of bytes the process directly read from disk
realtimeThe time required to compute the task
rssReal memory (resident set) size of the process
scratchFlag reporting the task was executed in a local scratch path
scriptThe task command script
startTask execution start timestamp
statusThe task execution status
submitTask submission timestamp
syscrNumber of read-like system call invocations that the process performed
syscwNumber of write-like system call invocations that the process performed
tagNextflow tag associated to the task execution
task_idNextflow task ID
timeTask execution timeout requested
vmemVirtual memory size used by the task execution
vol_ctxtNumber of voluntary context switches
wcharNumber of bytes the process wrote, using any write-like system call
workdirTask execution work directory
write_bytesNumber of bytes the process written to disk