Skip to content

fix: hold spill temp files by path instead of open file descriptor - #27602

Open
adriangb wants to merge 1 commit into
apache:mainfrom
adriangb:fix/spill-file-hold-path-not-fd
Open

adriangb wants to merge 1 commit into
apache:mainfrom
adriangb:fix/spill-file-hold-path-not-fd

Conversation

@adriangb

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Each RefCountedTempFile currently holds an Arc<NamedTempFile>, which keeps an open OS file descriptor for the entire lifetime of the spill file even while it sits idle on disk.

Under tight memory limits (e.g., 256 MB–512 MB with 16 partitions), operators that accumulate many spill runs—such as a hash aggregate before its final merge—hold one file descriptor per live spill file. On ClickBench Q18 at a 256 MB limit, this reaches ~1,000 live runs and fails with Too many open files (EMFILE) under the default ulimit -n of 1024.

RefCountedTempFile::read_stream already opens the file by path on demand. Holding TempPath instead of NamedTempFile means live spill files hold no file descriptors, so descriptor usage scales with active open writers and readers rather than total live spill files.

What changes are included in this PR?

  • Store Arc<TempPath> instead of Arc<NamedTempFile> in RefCountedTempFile (datafusion/execution/src/disk_manager.rs), converting the created file via .into_temp_path().
  • Open the path on demand in OpenOptions::new().append(true) mode in RefCountedTempFile::open_writer, preserving shared-offset append behavior across multiple writers on the same spill file.
  • Remove RefCountedTempFile::inner(&self) -> &NamedTempFile (unused in the workspace).
  • Add a Unix unit test live_spill_files_hold_no_file_descriptors verifying that 400 live spill files do not accumulate open file descriptors and still read back and clean up on drop.

Are these changes tested?

  • cargo test -p datafusion-execution --lib (including the new live_spill_files_hold_no_file_descriptors test)
  • cargo clippy -p datafusion-execution --all-targets -- -D warnings
  • cargo fmt --all -- --check

Are there any user-facing changes?

Breaking public API change: RefCountedTempFile::inner(&self) -> &NamedTempFile is removed because RefCountedTempFile no longer holds an open NamedTempFile. Callers that need the path can use RefCountedTempFile::path(), and callers that need to write or read can use the SpillFile trait methods.

Each RefCountedTempFile kept a NamedTempFile—and with it one open file descriptor—for the entire lifetime of the spill file. Operators that retain many spill runs (such as a hash aggregate before its final merge) hold one descriptor per live run and can hit EMFILE (Too many open files) under tight memory limits with the default ulimit -n of 1024.

Store TempPath in RefCountedTempFile instead and open the file on demand in append mode in open_writer (read_stream already opens the path on demand). Live spill files then hold no file descriptors, so descriptor usage scales with active writers and readers instead of total live spill files.

Removes RefCountedTempFile::inner(&self) -> &NamedTempFile, which is unused in the workspace.
@github-actions github-actions Bot added the execution Related to the execution crate label Nov 20, 2026
@adriangb
adriangb marked this pull request as ready for review November 20, 2026 06:04
files.push(file);
}

let held = open_fd_count().saturating_sub(before);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if other tests are opening files?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

execution Related to the execution crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant