Skip to content

fix: writeAtomically fails for file names longer than 217 bytes - #20493

Merged
FrankChen021 merged 1 commit into
apache:masterfrom
RiccardoSale:fix-write-atomically-long-file-name
Oct 8, 2026
Merged

FrankChen021 merged 1 commit into
apache:masterfrom
RiccardoSale:fix-write-atomically-long-file-name

Conversation

@RiccardoSale

@RiccardoSale RiccardoSale commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #20492.

Description

FileUtils.writeAtomically writes to a temporary file named .<fileName>.<uuid> before moving it to the target. The temporary name adds 38 bytes. On filesystems with a filename limit of 255 bytes, valid filenames of 218–255 bytes therefore cause the write to fail with File name too long.

WorkerTaskManager uses this method to persist assigned and completed tasks, with the task ID as the filename. Tasks with IDs in this range cannot start on a MiddleManager or Indexer.

Kill tasks reach this limit sooner because their IDs are longer than ingestion task IDs for the same datasource:

Task ID format Characters added to the datasource name
Kafka ingestion index_kafka_<dataSource>_<hash>_<random> 37
Batch ingestion index_parallel_<dataSource>_<random>_<createdTime> 49
Coordinator kill coordinator-issued_kill_<dataSource>_<random>_<intervalStart>_<intervalEnd>_<createdTime> 108

As a result, a datasource can ingest successfully while its kill tasks consistently fail to start.

Changes

  • Use .<uuid> as the temporary filename so its length is independent of the target filename. The UUID preserves uniqueness, and the target retains its full name after the move. This allows task IDs up to the filesystem limit of 255 bytes.
  • Add tests covering a filename of 255 characters and assignment of a task with an ID of 255 characters through WorkerTaskManager.

Proposed follow-up: kill task IDs without the interval

After this fix, kill tasks still support shorter datasource names than ingestion tasks: 147 characters, compared with 206 for batch ingestion and 218 for Kafka ingestion.
Would it make sense to aim for parity between ingestion and kill tasks in the datasource name lengths they support, so that any datasource that can be ingested can also be cleaned up?

The interval is already stored in the task payload.

Release note

Fixed: Tasks with IDs longer than 217 characters no longer fail to start on MiddleManagers and Indexers with File name too long.


Key changed/added classes in this PR
  • FileUtils
  • FileUtilsTest
  • WorkerTaskManagerTest

This PR has:

  • been self-reviewed.
  • a release note entry in the PR description.
  • added or updated unit tests.

FileUtils.writeAtomically named its temporary file ".<fileName>.<uuid>",
adding 38 characters to the target name. File names between 218 and 255
bytes were valid, but their temporary names were not, so the write failed
with "File name too long". WorkerTaskManager persists tasks this way using
the task ID as the file name, so tasks with such IDs, such as kill tasks
on datasources with long names, could never start.

Name the temporary file ".<uuid>" instead. The UUID keeps it unique, and
the target file keeps its full name after the move.

Fixes apache#20492.

@RiccardoSale RiccardoSale left a comment •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self review.

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

Reviewed all three changed files in the prepared diff:

  • indexing-service/src/test/java/org/apache/druid/indexing/worker/WorkerTaskManagerTest.java
  • processing/src/main/java/org/apache/druid/java/util/common/FileUtils.java
  • processing/src/test/java/org/apache/druid/java/util/common/FileUtilsTest.java

The filename-independent temporary name preserves the existing CREATE_NEW, write/fsync, atomic replacement, and cleanup flow while allowing targets at the filesystem component-length limit. I traced task assignment and completion persistence and searched the writeAtomically call sites; no affected behavior or compatibility issue surfaced.

Validation: git diff --check 87525a9c6d50474efc94cbc01706fa28c99b1dc1...HEAD passed. Static review only; no tests were run.


This is an automated review by Codex GPT-5.6-Luna(max)

@FrankChen021
FrankChen021 merged commit 4ad1ddd into apache:master Oct 8, 2026
27 of 28 checks passed
@github-actions github-actions Bot added this to the 39.0.0 milestone Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tasks with IDs longer than 217 bytes fail on MiddleManager/Indexer with File name too long

2 participants