Getting started with data lineage
Data lineage in Nextflow provides comprehensive tracking of workflow runs, task executions, and output files. This feature helps you verify the integrity and reproducibility of your pipeline results by maintaining a complete history of computations and intermediate data.
Data lineage is an experimental feature added in Nextflow 25.04. The functionality may change in future releases.
Overview
Data lineage tracks the complete history of your Nextflow runs, including:
- Workflow runs and their configurations
- Task executions and their inputs/outputs
- File outputs and their provenance
Each lineage record has a unique identifier called a lineage ID (LID) that you can use to access and query the data.
The data model for every lineage record is defined in the Nextflow source code.
Enable data lineage
To enable data lineage tracking, add the following to your Nextflow configuration:
lineage.enabled = true
By default, lineage data is stored in the .lineage directory in your current working directory. You can customize this location:
lineage.store.location = '<PATH_TO_STORAGE>'
For global configuration, add these settings to $HOME/.nextflow/config.
See the lineage configuration scope for details.
Generate lineage metadata
Run a Nextflow pipeline to generate some lineage metadata. For example:
$ nextflow run rnaseq-nf -profile conda
Nextflow will automatically record the workflow run, task executions, and output files in the lineage store.
Explore lineage
Now that you have generated some lineage metadata, you can explore it from the command line using the lineage command. First, use the The lineage ID (LID) is the unique identifier for the workflow run and the entrypoint for exploring the lineage. Use the Every workflow run is represented in the lineage store as a The output files of a workflow run can be accessed as Files must be published to the run's output directory ( List the output directory to see the available files: Now, use the workflow LID and relative path to view the lineage record for an output file: Every output file is represented in the lineage store as a As this record is a workflow output, it is not linked directly to a task run. Instead, it is linked to the original task output. The Any LID in a lineage record can be viewed, allowing you to traverse the lineage metadata interactively. Use the value of This record is the task output for the same file -- it has a value for View the lineage record for the task that produced this file: Every task run is represented in the lineage store as a The previous example traced a workflow output to its producing task. Conversely, the outputs of a task can be retrieved with the Each entry is one declared task output. An agent also runs as a task, but its provenance is a model and prompt rather than a script. Agent executions are therefore recorded as a separate An agent's results are recorded as a See Data lineage in the agent documentation for the full list of what is and is not captured. Finally, use the Open the HTML report in a web browser to view the lineage graph.Workflow runs
list subcommand to list the workflow runs in the lineage store:$ nextflow lineage list
TIMESTAMP RUN NAME SESSION ID LINEAGE ID
2025-05-09 13:28:30 CDT peaceful_blackwell 065bdc6b-89b4-42ee-92c1-2a5af37f2c50 lid://16b31030474f2e96c55f4940bca3ab64view subcommand to view the lineage record for the workflow run:$ nextflow lineage view lid://16b31030474f2e96c55f4940bca3ab64
{
"version": "lineage/v1beta1",
"kind": "WorkflowRun",
"spec": {
"workflow": {
"scriptFiles": [
...
],
"repository": "https://github.com/nextflow-io/rnaseq-nf",
"commitId": "86165b8c81d43a1f57363964431395152e353e56"
},
"sessionId": "065bdc6b-89b4-42ee-92c1-2a5af37f2c50",
"name": "peaceful_blackwell",
"params": [
...
],
"config": {
...
},
"metadata": {
...
}
}
}WorkflowRun record. It includes information such as the pipeline repository, revision, run name, parameters, resolved config, and run metadata.Workflow outputs
lid://<WORKFLOW_RUN_HASH>/<PATH>, where <PATH> is the file path relative to the workflow output directory.outputDir config option or -output-dir command line option) in order to be recorded as workflow outputs in the lineage store.$ find results
results
results/fastqc_ggal_gut_logs
results/fastqc_ggal_gut_logs/ggal_gut_1_fastqc.html
results/fastqc_ggal_gut_logs/ggal_gut_1_fastqc.zip
results/fastqc_ggal_gut_logs/ggal_gut_2_fastqc.html
results/fastqc_ggal_gut_logs/ggal_gut_2_fastqc.zip
results/multiqc_report.html$ nextflow lineage view lid://16b31030474f2e96c55f4940bca3ab64/multiqc_report.html
{
"version": "lineage/v1beta1",
"kind": "FileOutput",
"spec": {
"path": "/results/multiqc_report.html",
"checksum": {
"value": "03fd5ed150c7862e1fad5efd4f574a47",
"algorithm": "nextflow",
"mode": "standard"
},
"source": "lid://862df53160e07cd823c0c3960545e747/multiqc_report.html",
"workflowRun": "lid://16b31030474f2e96c55f4940bca3ab64",
"taskRun": null,
"size": 5079806,
"createdAt": "2025-05-09T13:27:34.576590545-05:00",
"modifiedAt": "2025-05-09T13:27:34.586590551-05:00",
"labels": null
}
}FileOutput record. It includes basic file information, such as the real path, checksum, file size and created/modified timestamps, as well as lineage information, such as the workflow run and task run that produced it.labels field is null because no labels were assigned to this file. Labels are set using the label directive in the output block.source to view the original task output:$ nextflow lineage view lid://862df53160e07cd823c0c3960545e747/multiqc_report.html
{
"version": "lineage/v1beta1",
"kind": "FileOutput",
"spec": {
"path": "/work/86/2df53160e07cd823c0c3960545e747/multiqc_report.html",
"checksum": {
"value": "b14f5171a48ce5c22ea27d7b8e57b6c4",
"algorithm": "nextflow",
"mode": "standard"
},
"source": "lid://862df53160e07cd823c0c3960545e747",
"workflowRun": "lid://16b31030474f2e96c55f4940bca3ab64",
"taskRun": "lid://862df53160e07cd823c0c3960545e747",
"size": 5079806,
"createdAt": "2025-05-09T13:27:34.236590379-05:00",
"modifiedAt": "2025-05-09T13:27:34.246590383-05:00",
"labels": null
}
}taskRun which is the same as its source.Task runs
$ nextflow lineage view lid://862df53160e07cd823c0c3960545e747
{
"version": "lineage/v1beta1",
"kind": "TaskRun",
"spec": {
"sessionId": "065bdc6b-89b4-42ee-92c1-2a5af37f2c50",
"name": "MULTIQC",
"codeChecksum": {
"value": "edf2e9f84cd3a18ee9259012b660f2dd",
"algorithm": "nextflow",
"mode": "standard"
},
"script": "\n cp multiqc/* .\n echo \"custom_logo: $PWD/nextflow_logo.png\" \u003e\u003e multiqc_config.yaml\n multiqc -n multiqc_report.html .\n ",
"eval": null,
"input": [
{
"type": "path",
"name": "*",
"value": [
"lid://eff8846883b46c5a76f11e7e4480a6c8/ggal_gut",
"lid://2d8bd92c69f732605bc99941e60d5319/fastqc_ggal_gut_logs"
]
},
{
"type": "path",
"name": "config",
"value": [
{
"path": "https://github.com/nextflow-io/rnaseq-nf/tree/86165b8c81d43a1f57363964431395152e353e56/multiqc",
"checksum": {
"value": "2aac500cdfb292e961e678433e7dc3d8",
"algorithm": "nextflow",
"mode": "standard"
}
}
]
}
],
"container": null,
"conda": "file:///conda/env-4a436c230263dfdbbf4dddd0623505d1",
"spack": null,
"architecture": null,
"globalVars": {},
"binEntries": [],
"workflowRun": "lid://16b31030474f2e96c55f4940bca3ab64",
"moduleId": null
}
}TaskRun, which includes information such as the name, script, inputs, and software dependencies. From here, you can continue traversing through the file inputs to view upstream tasks.Task outputs
#output fragment of the task LID. For example, view the outputs of the QUANT task that produced one of the inputs shown earlier:$ nextflow lineage view lid://eff8846883b46c5a76f11e7e4480a6c8#output
[
{
"type": "val",
"name": "id",
"value": "ggal_gut"
},
{
"type": "path",
"name": null,
"value": "lid://eff8846883b46c5a76f11e7e4480a6c8/quant_ggal_gut"
}
]type is the output qualifier (path, val, env, eval, or stdout), name is the output name (null when the output declares only a file pattern), and value is a lid:// reference for file outputs or the literal value otherwise.Agent runs
AgentRun record in place of a TaskRun:$ nextflow lineage view lid://c47bf9183c56715c9bca1a67a4acdc68
{
"version": "lineage/v1beta1",
"kind": "AgentRun",
"spec": {
"sessionId": "554fe81b-8034-4f5a-81c4-b07195258201",
"name": "analyst (1)",
"codeChecksum": {
"value": "f8acadb0cd9048eaf953b1b30836dffd",
"algorithm": "nextflow",
"mode": "standard"
},
"runner": "pi",
"model": "openai/gpt-5-mini",
"resolvedModel": null,
"instruction": "You are a precise scientific analyst. Be concise and honest about uncertainty.",
"goal": null,
"promptTemplate": " \"\"\"\n Analyze the following question and return a structured analysis.\n\n Question: ${query.question}\n \"\"\"\n",
"maxIterations": 20,
"outputSchema": "{\"additionalProperties\":false,\"properties\":{\"actionable\":{\"type\":\"boolean\"},\"confidence\":{\"type\":\"number\"},\"key_points\":{\"items\":{\"type\":\"string\"},\"type\":\"array\"},\"summary\":{\"type\":\"string\"}},\"required\":[\"summary\",\"confidence\",\"actionable\",\"key_points\"],\"type\":\"object\"}",
"tools": null,
"skills": null,
"input": [
{
"type": "val",
"name": "query",
"value": {
"question": "Is FASTQ a binary or a text format?",
"context": "bioinformatics file formats"
}
}
],
"container": null,
"workflowRun": "lid://6334982d0dd5e6573989fa5640fc01d3",
"moduleId": null
}
}TaskOutput, exactly like a process, so agent outputs traverse and query the same way as any other task output.Lineage graph
render subcommand to render the entire lineage of the MULTIQC report as an HTML report:$ nextflow lineage render lid://16b31030474f2e96c55f4940bca3ab64/multiqc_report.html
Rendered lineage graph for lid://16b31030474f2e96c55f4940bca3ab64/multiqc_report.html to lineage.html
Query lineage records
To find a lineage record, you normally have to know the LID of the record or a downstream record (such as a workflow run) from which you can traverse to the desired record. However, you can also query the entire lineage store by fields to quickly find relevant records and aggregate records from different runs.
Use the find subcommand to find all tasks executed by a workflow run:
$ nextflow lineage find type=TaskRun workflowRun=lid://16b31030474f2e96c55f4940bca3ab64
lid://2d8bd92c69f732605bc99941e60d5319
lid://eff8846883b46c5a76f11e7e4480a6c8
lid://862df53160e07cd823c0c3960545e747
lid://6d3bff36bf2c3c14c2d383384621e8ca
You can use any field defined in the lineage data model. Fields specific to a record kind work the same way -- for example, to find every agent run in the store that used a given model:
$ nextflow lineage find type=AgentRun model=openai/gpt-5-mini
lid://c47bf9183c56715c9bca1a67a4acdc68
The view subcommand outputs JSON, so you can use JSON processing tools such as jq to further query and transform results. The find subcommand prints one LID per line, which pipes directly into xargs.
Use lineage with resume
Resumed runs are recorded as distinct runs that share a session ID.
Every run writes a new WorkflowRun record with its own LID, including resumed runs, so nextflow lineage list shows one row per run:
$ nextflow lineage list
TIMESTAMP RUN NAME SESSION ID LINEAGE ID
2025-05-09 13:28:30 CDT peaceful_blackwell 065bdc6b-89b4-42ee-92c1-2a5af37f2c50 lid://16b31030474f2e96c55f4940bca3ab64
2025-05-09 14:02:11 CDT wise_euler 065bdc6b-89b4-42ee-92c1-2a5af37f2c50 lid://65044872aad36f97e42336b9ba0dee57
Both rows have the same session ID because -resume reuses the session ID of the initial run. The session ID is recorded in the sessionId field of the WorkflowRun record, as well as every TaskRun record, which makes it the key for querying an entire resume chain.
A cached task is not re-executed, so it does not produce a new TaskRun record. Its LID is the task hash, so the record written by the run that originally executed the task is still the record for that task. The workflowRun field of a TaskRun therefore points to the run that actually executed it, not necessarily the resumed run.
This distinction matters when querying. Finding tasks by workflowRun returns only the tasks that the given run executed:
$ nextflow lineage find type=TaskRun workflowRun=lid://65044872aad36f97e42336b9ba0dee57
Finding tasks by sessionId returns every task across the initial run and all of its resumes:
$ nextflow lineage find type=TaskRun sessionId=065bdc6b-89b4-42ee-92c1-2a5af37f2c50
Workflow outputs behave differently from task runs. Nextflow publishes the outputs of cached tasks on every run, so each run gets its own FileOutput records under its own LID, and the source field of each one points back to the task output that produced it. A resumed run's output directory is fully described by its own WorkflowRun LID, whether or not its tasks were cached.
Compare task runs
Task run LIDs are based on the standard task hash, which makes it easy to compare two task runs in the lineage metadata. For example, if a task is unexpectedly re-executed during a resumed run, as long as lineage is enabled for both the initial and resumed runs, the two tasks can be compared without any additional runs.
This section builds on the above rnaseq-nf example to demonstrate how to compare two task runs in the event of a cache invalidation.
First, modify the pipeline in a way that invalidates the cache for the MULTIQC process. For example, modify the process script.
Resume the pipeline. It will re-execute the MULTIQC process:
$ nextflow run rnaseq-nf -profile conda -resume
...
[6d/3bff36] process > RNASEQ:INDEX (ggal_1_48850000_49020000) [100%] 1 of 1, cached: 1 ✔
[2d/8bd92c] process > RNASEQ:FASTQC (FASTQC on ggal_gut) [100%] 1 of 1, cached: 1 ✔
[ef/f88468] process > RNASEQ:QUANT (ggal_gut) [100%] 1 of 1, cached: 1 ✔
[94/33dda7] process > MULTIQC [100%] 1 of 1 ✔
Retrieve the hash of the MULTIQC run from the log file or work directory. Compare it to the task hash of the initial run:
$ nextflow lineage diff lid://862df53160e07cd823c0c3960545e747 lid://9433dda73f2193491f9a26e3e23cd8a1
diff --git 862df53160e07cd823c0c3960545e747 9433dda73f2193491f9a26e3e23cd8a1
--- 862df53160e07cd823c0c3960545e747
+++ 9433dda73f2193491f9a26e3e23cd8a1
@@ -4,11 +4,11 @@
"sessionId": "065bdc6b-89b4-42ee-92c1-2a5af37f2c50",
"name": "MULTIQC",
"codeChecksum": {
- "value": "edf2e9f84cd3a18ee9259012b660f2dd",
+ "value": "9615a8da3a3f9e935cfc8e4042cdf5e0",
"algorithm": "nextflow",
"mode": "standard"
},
- "script": "\n cp multiqc/* .\n echo \"custom_logo: $PWD/nextflow_logo.png\" \u003e\u003e multiqc_config.yaml\n multiqc -n multiqc_report.html .\n ",
+ "script": "\n cp multiqc/* . # hello!\n echo \"custom_logo: $PWD/nextflow_logo.png\" \u003e\u003e multiqc_config.yaml\n multiqc -n multiqc_report.html .\n ",
"eval": null,
"input": [
{
@@ -40,7 +40,7 @@
"architecture": null,
"globalVars": {},
"binEntries": [],
- "workflowRun": "lid://16b31030474f2e96c55f4940bca3ab64",
+ "workflowRun": "lid://65044872aad36f97e42336b9ba0dee57",
"moduleId": null
}
}
Note the difference between the task scripts, highlighting the change that caused the task to be re-executed.
Use lineage with workflow outputs
Workflow outputs declared in the output block are also recorded in the lineage store. The output of a workflow run is accessible as lid://<WORKFLOW_RUN_HASH>#output.
For example, run the rnaseq-nf pipeline with the preview-25-04 branch, which uses the output block to publish outputs:
$ nextflow run rnaseq-nf -r preview-25-04 -profile conda
View the workflow output in the lineage metadata:
$ nextflow lineage view lid://9410d13abeec617640b5fe9735ba12fc#output
[
{
"type": "Collection",
"name": "samples",
"value": "lid://9410d13abeec617640b5fe9735ba12fc/samples.json"
},
{
"type": "Path",
"name": "summary",
"value": "lid://9410d13abeec617640b5fe9735ba12fc/multiqc_report.html"
}
]
This record is a WorkflowOutput record. It provides a structured view of all published output files, allowing you to traverse them directly from the workflow run.
Nextflow types are represented in workflow outputs as follows:
| Type | JSON representation | Nextflow type |
|---|---|---|
Collection | array (or index file path) | Bag, List, Set |
Map | object | Map, Record |
Path | string with lid:// prefix | Path |
Output labels
When labels are assigned to a workflow output with the label directive, they appear in the labels field of each corresponding FileOutput record:
$ nextflow lineage view lid://9410d13abeec617640b5fe9735ba12fc/multiqc_report.html
{
"version": "lineage/v1beta1",
"kind": "FileOutput",
"spec": {
"path": "/results/multiqc_report.html",
...
"labels": ["qc", "summary"]
}
}
Labels can be used to filter files when querying lineage records with the nextflow lineage find command.
See Labels for details on assigning labels to workflow outputs.
Use lineage with publishDir
Lineage does not require the A published file is recorded in lineage only if it lands inside the run's output directory ( Set A run without an The Start with the published file. Follow its The output block. It is possible to produce lineage with the publishDir directive, but it requires a few extra steps.Set output directory
outputDir), which defaults to results. Nextflow resolves a publishDir path against the launch directory, not against outputDir, so the two settings are not guaranteed to match.outputDir to a directory that contains every publishDir target:outputDir = params.outdirFind workflow outputs by publish path
output block produces no WorkflowOutput record, so lid://<WORKFLOW_RUN_HASH>#output returns nothing. Look up published files by their path relative to the output directory instead, as shown in Workflow outputs:$ nextflow lineage view lid://16b31030474f2e96c55f4940bca3ab64/multiqc_report.htmlRecover output metadata
output block can publish metadata alongside files, while publishDir only publishes files. However, this metadata can be retrieved by tracing a published file back to the producing task and reading that task's outputs.source to the task run, then view the task output, as shown in Task runs and Task outputs:$ nextflow lineage view lid://<TASK_RUN_HASH>#outputTaskOutput record holds every declared output, not just files, so any metadata emitted as a val output is recorded there.
Use lineage in a Nextflow script
Since lineage IDs are valid URIs, output files in the lineage store can be accessed by their LID in a Nextflow script, like any other path. The LID path returns the real path as defined by the path field in the FileOutput record.
The following script uses the samples.json from the previous example as an input samplesheet:
channel.fromPath('lid://9410d13abeec617640b5fe9735ba12fc/samples.json')
.splitJson()
.view()
It should produce the following output:
[id:gut, quant:/results/gut/quant, fastqc:/results/gut/fastqc]
The fromLineage channel factory can also be used to query lineage records in a similar manner as the find subcommand. See fromLineage for details.