51 posts tagged with "fusion"

Fusion v2.6.4 adds retryable and non-retryable exit codes for startup failures in NFS service mode, and new health and liveness metrics. It also fixes renamed directories and very large files on Google Cloud Storage.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.6.4-amd64.json'
}

Feature updates and improvements​

NFS service mode​

• Added distinct exit codes for startup failures when Fusion runs as the NFS daemon and cannot complete its startup handshake with the control node. Fusion exits with 172 when the control node rejects the request, which is a configuration or permission problem that a retry cannot fix. Fusion exits with 173 for anything else, such as an unreachable or slow control node, and for ambiguous failures. Intelligent Compute relocates the task on 173 instead of aborting the run. See Exit codes 172 and 173.
• Added FUSION_NFS_READINESS_TIMEOUT to set how long Fusion waits for its work directory to become ready before it classifies the failure and exits. The default is 60 seconds.

Metrics​

• Added the fusion_health_* metric family to the per-task metrics file. It reports uptime, memory use split into non-overlapping categories, open file handles, cache device disk usage, mount initialization time, and shutdown flush time. Resource exhaustion is now visible after the fact.
• Added two liveness gauges that report how many file system requests are outstanding and the age of the oldest one. The age stays near zero when nothing is calling Fusion and climbs when Fusion has taken a request and not answered it. Use them to tell a stalled mount from an idle one.
• Published a schema and triage runbook for the metrics file. See Fusion task metrics.

Bug fixes​

General​

• Fixed a renamed directory dropping files and symlinks that were not yet uploaded.
• Fixed a "no such file" error on NFS daemon clusters for an input produced by another task.
• Fixed a retried task starting on leftovers from the previous attempt.
• Fixed .tbi, .csi, and .bai index files appearing older than the data they index.
• Fixed symlink targets longer than 2048 bytes being truncated on read.
• Fixed declared outputs with adjacent brace patterns being evicted at shutdown instead of persisted.
• Fixed a truncate or seek-write to an implausible offset exhausting memory and killing the mount, which is now rejected with a clear error.
• Fixed temporary files being uploaded while the task was still writing them, except on runs with snapshots enabled.

Google Cloud​

• Fixed corrupt results for very large files on Google Cloud Storage, roughly 128 GiB and above at the default chunk size.
• Added a check that rejects an object over the 5 TiB Google Cloud Storage per-object limit before uploading.

Dependencies​

• Reorganized the metrics collector, added a streaming upload endpoint on the control interface, and moved fusion-tracer onto it.
• Updated CI, build pipeline, and dependencies.

Upgrade notes​

• Drop-in replacement for v2.6.3. No configuration changes required.
• In NFS mode, fusion-tracer now sends its trace through the control interface and no longer falls back to writing it through the mount. A trace produced while the control node is unreachable is dropped. Tracing is unchanged when the control node is reachable.
• FUSION_NFS_READINESS_TIMEOUT is new and defaults to 60 seconds.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.5.14-amd64.json'
}

Bug fixes​

General​

• Fixed data loss when a file move is interrupted.
• Fixed declared outputs with brace patterns such as *.{bcf,vcf}{,.gz} not being saved.
• Fixed symbolic links not persisting their backing data.
• Fixed long symbolic link targets being truncated on read.
• Fixed temporary task files being uploaded while still being written.
• Fixed temporary working files being left behind after a retried operation.
• Fixed the advertised file system block size, which is now 1 MiB instead of 512 bytes.
• Fixed an internal cache maintenance task continuing to run after shutdown.
• Fixed an internal inode allocation check returning an identifier that was not free.

Google Cloud​

• Fixed corrupted results when assembling very large files on Google Cloud Storage.

Dependencies​

• Hardened internal tests against data races and a timing-related flake.
• Updated dependencies and GitHub Actions.

Upgrade notes​

• Drop-in replacement for v2.5.13. No configuration changes required.

Fusion v2.6.3 fixes a mount failure for task images with a non-root USER that mount the Fusion NFS export through the setuid helper.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.6.3-amd64.json'
}

Bug fixes​

NFS service mode​

• Fixed mounts failing with exit code 174 when invoked through the setuid fusion-suid helper.

Upgrade notes​

• Drop-in replacement for v2.6.2. No configuration changes required.

Fusion v2.6.1 adds per-backend request metrics for AWS S3, Azure Blob Storage, and Google Cloud Storage, and fixes uploads, mounts, reads, symlink metadata, and the local file system backend.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.6.1-amd64.json'
}

Feature updates and improvements​

Metrics​

• Added request metrics for every cloud backend. Fusion records latency, retries, error codes, and outcomes for each request to AWS S3, Azure Blob Storage, and Google Cloud Storage. New metric families cover file system operations, the content cache, and chunked I/O, with standardized names and a per-routine label. Fusion writes the metrics as a compact multi-member gzip file by default. See Fusion task metrics.

General​

• Added a control interface for direct uploads. Studios and fusion-tracer push data to cloud storage without going through the mount, and a task can upload while it is still running. fusion-tracer now uploads trace.json through this path.

Google Cloud​

• Added environment variables to tune the retry backoff timing for Google Cloud Storage.

Bug fixes​

General​

• Fixed a moved file removing its source before the copy completed.
• Fixed the inode allocator handing out an inode that was not verified free.
• Fixed a misleading "is a directory" error, now "permission denied", when opening a bucket you cannot read.
• Fixed a "no such file" error when a task reads a symlink staged by a previous task on Nextflow 24.04 and later.
• Fixed very large writes failing on oversized upload parts on AWS S3 and Azure Blob Storage.
• Fixed throttled mounts timing out during initialization.
• Fixed read operations not releasing their network connections.
• Fixed truncation and full-content writes on the POSIX backend.
• Fixed chunk cleanup failures being swallowed.
• Changed the sync task queue to drop from the tail and skip cancelled tasks.
• Fixed the TTL cache janitor not stopping on shutdown.
• Fixed the reported st_blksize hint, which now advertises a 1 MiB preferred I/O block size.

Google Cloud​

• Added automatic retries for transient Google Cloud Storage errors.

Dependencies​

• Updated the Go toolchain to 1.26.5, fusion-sdk to v0.2.1, and go-fuse to v2.10.1.
• Updated CI and build dependencies.

Upgrade notes​

• Drop-in replacement for v2.6.0. No configuration changes required.
• Fusion now stores the metrics file as gzip by default. If you read the metrics file directly, decompress it first.
• If you rely on fusion-tracer output, skip this release and use v2.6.2.

Fusion v2.6.2 fixes a regression introduced in v2.6.1 where fusion-tracer could stop producing output.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.6.2-amd64.json'
}

Bug fixes​

fusion-tracer​

• Fixed fusion-tracer producing no trace.json when the control node is unreachable by falling back to writing the trace through the mount.

Upgrade notes​

• Drop-in replacement for v2.6.1. No configuration changes required.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.5.13-amd64.json'
}

Feature updates and improvements​

General​

• Added a control interface for direct uploads. Studios and fusion-tracer can push data to cloud storage without going through the mounted file system. A task can now upload data reliably while it is still running.
• Improved failure reporting when the Fusion file system process exits before the mount completes. Fusion now reports the failure immediately instead of entering a retry loop.

Google Cloud​

• Added environment variables to tune the retry backoff timing for Google Cloud Storage.

Bug fixes​

General​

• Fixed silent write loss on mounts without write permission by verifying write access at startup.
• Fixed a misleading "is a directory" error, now "permission denied", when opening a bucket you do not have permission to read.
• Fixed large file uploads failing when a write exceeded the cloud provider maximum upload part size on AWS S3 and Azure Blob Storage.
• Fixed interrupted uploads leaking temporary staging data into workflow output folders.
• Fixed mounts timing out under heavy rate limiting at startup.
• Fixed read operations not releasing their network connections.
• Fixed errors during file data cleanup being hidden.

Google Cloud​

• Added automatic retries for transient Google Cloud Storage errors.

Dependencies​

• Updated dependencies.

Upgrade notes​

• Drop-in replacement for v2.5.12. No configuration changes required.

Fusion v2.6.0 adds NFS service mode, the Fusion Doctor diagnostics tool, new metrics, and cloud storage reliability improvements.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.6.0-amd64.json'
}

Feature updates and improvements​

NFS service mode​

• Added NFS service mode. Fusion runs as a long-lived daemon and serves the mount over NFS, instead of one FUSE mount per process. A single Fusion instance backs many tasks and isolates metadata per task. An HTTP control service registers, flushes, and cleans up each work directory. Snapshots and work directory cleanup now survive Spot reclamation.

Fusion Doctor​

• Added the fusion doctor subcommand. It validates the runtime environment and reports common mount, configuration, and connectivity problems with remediation steps. It supports custom thresholds and profile validation, and an opt-in --redact flag strips personal data from reports. Fusion Doctor also ships as a standalone binary and container image for use outside a running mount. See Fusion Doctor.

Storage​

• Added support for prefix-restricted S3 bucket policies, and a fallback chain for S3 bucket region detection.
• Changed archived Azure blobs to map to a dedicated "resource archived" error, and fixed the Azure CopyFromURL size threshold.
• Improved the classification and messages for non-recoverable I/O errors.
• Improved chunk download resilience on unreliable networks. Reads no longer fail with EIO when an object store stream stalls or truncates mid-transfer. Fusion watches each chunk download for progress, cancels stalled streams, resumes from where the transfer left off, and caps downloads with a configurable timeout.

Spot instances​

• Removed the dependency on fusermount. Fusion exits cleanly on Spot reclamation instead of failing with EINVAL, which reduces orphaned mounts and lost work.

Metrics​

• Added OpenTelemetry metrics for read and write operations, with file output and JSON formatting.
• Added process metrics collection to fusion-tracer, written to trace.json. The GPU tracer is enabled by default.

Fusion Snapshots​

• Enabled incremental snapshot dumps by default.
• Updated fusion-snapshot to 1.9.0, with symbolic link remapping in NFS mode.

Dependencies​

• Updated dependencies.

Upgrade notes​

• Drop-in replacement for v2.5.x. No configuration changes required.
• NFS service mode applies only to Intelligent Compute. Per-task FUSE mounts on other compute environments are unaffected.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.5.12-amd64.json'
}

Feature updates and improvements​

General​

• Improved chunk download resilience on unreliable networks. Reads no longer fail with EIO when an object store stream stalls or truncates mid-transfer. Fusion watches each chunk download for progress, cancels streams that stop, and resumes from where the transfer left off with exponential backoff. This removes the multi-minute hangs and read failures seen in environments with intermittent object store connectivity, typically missing VPC endpoints or congested egress paths. Healthy slow links are unaffected.
• Added a log entry when an application tries to create a hard link inside a Fusion mount, for example certain bioinformatics tools and snapshot workflows. The entry names the operation and the calling process and points to the available workarounds, a symbolic link or a copy. Hard links remain unsupported and the error returned to the application is unchanged.

Dependencies​

• Updated dependencies.

Upgrade notes​

• Drop-in replacement for v2.5.x. No configuration changes required.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.4.22-amd64.json'
}

Bug fixes​

Azure​

• Fixed symlink compaction failing with HTTP 409 CannotVerifyCopySource errors for files between 256 MiB and 4 GiB.
• Changed reads of Azure Archive tier blobs to fail immediately instead of being retried.

General​

• Fixed stale symlink content being served after a failed compaction.
• Changed cloud error handling to retry only transient errors and fail fast on permanent errors such as forbidden access and invalid credentials.
• Added a structured warning when an application calls the unsupported link(2) inside a Fusion mount.

Dependencies​

• Updated build dependencies.

Upgrade notes​

• Drop-in replacement for v2.4.x. No configuration changes required.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.5.11-amd64.json'
}

Bug fixes​

AWS​

• Added support for prefix-restricted S3 bucket policies.
• Improved S3 bucket region detection with a fallback chain instead of defaulting to eu-west-1.

Azure​

• Changed reads of an Azure Archive tier blob to return a "resource archived" error instead of a generic cloud provider error.
• Fixed symlink compaction failing for files between 256 MiB and 4 GiB.

General​

• Removed an unnecessary truncate before each append-mode write.
• Fixed a NoSuchKey crash when an application renames a file and immediately creates a new one at the same path.
• Fixed partial chunk uploads producing incomplete or corrupted objects.
• Fixed symlinks being served as a short path string instead of the file content when symlink compaction failed.

fusion-tracer​

• Removed spurious Failed to fetch entry size and Failed to load file metadata errors.
• Built fusion-tracer against glibc 2.17 so it runs on container images with glibc 2.17 or later and loads the NVIDIA management library on GPU nodes.

Dependencies​

• Updated CI and build dependencies.

Upgrade notes​

• Drop-in replacement for v2.5.x. No configuration changes required.