Skip to content

Stop preprocess deleting the interim directory - #220

Merged
celefthe merged 4 commits into
mainfrom
celefthe/issue135
Oct 13, 2026
Merged

celefthe merged 4 commits into
mainfrom
celefthe/issue135

Conversation

@celefthe

Copy link
Copy Markdown
Member

Closes #135.

preprocess keeps its intermediate files in a fresh subdirectory of --interim_dir and removes only that subdirectory, whether it finishes, fails or stops early with --channel-means-only.

  • io.interim_workdir, a context manager that yields a temporary directory inside interim_dir. It creates interim_dir if missing and removes it afterwards only if it created it and it is left empty.
  • run_preprocessing runs everything from binning to the HDF5 write inside io.interim_workdir, replacing shutil.rmtree(interim_dir). Most of its diff is re-indentation.
  • Tests in tests/test_io.py and tests/test_preprocessing.py and a changelog entry.

@celefthe
celefthe merged commit c987ab2 into main Oct 13, 2026
8 checks passed
@celefthe
celefthe deleted the celefthe/issue135 branch October 13, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Preprocessing nukes itself if interim is same as output

1 participant