Skip to content

Add resume option - #339

Open
LouisLeNezet wants to merge 8 commits into
askimed:mainfrom
LouisLeNezet:test_resume
Open

LouisLeNezet wants to merge 8 commits into
askimed:mainfrom
LouisLeNezet:test_resume

Conversation

@LouisLeNezet

Copy link
Copy Markdown

This is a experimental PR to answer the following problematic:

When running test we often want to rerun the test without having to download back all the input data.
Sometimes we also just want to rerun the nf-test logic and not all the nextflow processing.

This PR introduce a new flag, that would have the same concept as the --resume of nextflow.
The aim is to not reset the test folder, and let nextflow do it's resume job by adding this flag to the execution.

I would like to go even further by putting all the input into a separate folder to allow it to be shared across test.
It is often the case that different test case use the same input file but with different configuration. This would then allow to reduce the memory used.

What do you think ?

@nttg8100

Copy link
Copy Markdown

Hi @LouisLeNezet , it is what I also think about. Can you add the workflow.nf.test with the new compiled nf-test so it can be verify directly instead of testing only on the java abstraction ?

@LouisLeNezet

Copy link
Copy Markdown
Author

Hi @nttg8100,

I've added a test and made sure that the workflow was indeed resumed.
This does seems to work.

@nttg8100

Copy link
Copy Markdown

Hi @nttg8100,

I've added a test and made sure that the workflow was indeed resumed. This does seems to work.

I tested it locally and found no error. I think the error may come from the specific version.

@LouisLeNezet

Copy link
Copy Markdown
Author

I've added the missing params.file initialisation for the NextflowCommandTest.

@LouisLeNezet

Copy link
Copy Markdown
Author

Hi,
Would it be possible to have a review @nttg8100 ?

@mahesh-panchal mahesh-panchal left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude review

I built the branch and ran it against Nextflow 26.10.3. The feature works, but I'd redesign it before merging. Three bugs come from adding a separate flag, and the new integration test can't pass.

Works

  • On a pipeline test and a process test with setup{}, run 1 shows cached=0 and run 2 shows cached=1 (cached=2 for the setup{} chain). Run 2 is faster.
  • A run without the flag still starts clean.

Suggested design: derive resume from options, drop the flag

Nextflow already has -resume, so I'd drop --dev-resume and have nf-test detect -resume in the test's options (from nf-test.config or a test file's options "..."). When it's present, keep the directories that make resume work.

  • This removes ITest.setDevResume, the engine and CLI plumbing, NextflowCommand.resume and Command.addEnvironment.
  • It fixes items 1 and 2 below by construction.
  • Match only the exact token -resume, not --resume, which is a pipeline parameter.
  • Log a debug line when it triggers.
  • A shared nf-test.config with -resume makes CI non-hermetic where work dirs persist, so the docs should recommend a local dev config via --config (not tested).
  • Anyone with options "-resume" today gets a different directory lifecycle after this, so it needs a changelog entry.

Must fix

  1. NXF_HOME is redirected to <launchDir>/.nxf-home (NextflowCommand.java:251).
    It has nothing to do with -resume. Plugins, secrets and assets are re-fetched for every test. Reproduction: plugins { id 'nf-schema@2.2.0' } with NXF_OFFLINE=true. Without the flag the test passes. With --dev-resume it fails with Plugin with id nf-schema not found in any repository. Remove the redirect.

  2. -resume is added unconditionally (NextflowCommand.java:184).
    With options "-resume" in nf-test.config plus --dev-resume, Nextflow fails with Can only specify option -resume once. This disappears with the options-based design.

  3. testWorkflowDevResume can never pass (WorkflowTest.java:224).

    • It asserts std.out contains "Stored process". In nf-test, std.out holds only the summary line, for example [SUCCESS] completed=0 failed=0 cached=1. Assert cached= is at least 1 instead.
    • findStdOutFile returns the first std.out it finds under .nf-test, which may belong to another test. It should read the one for the test that was run. It also returns null when none exists, which gives a null pointer.
    • The @BeforeAll to @BeforeEach change slows the whole class.
    • On my machine the test also fails because the fixture's tuple input is rejected by Nextflow 26.10.3. testRegex fails the same way on main, so that part isn't from this PR.

Should fix

  1. Resume with --update-snapshot can record stale snapshots.
    A task with an undeclared input (the script reads a file by path) is served from cache after the file changes. Reproduction: ext.txt changed from v1 to v2, then --dev-resume --update-snapshot kept v1 in the snapshot. A fresh run fails with Different Snapshot. Reject the combination, or warn.

  2. setupDirectory keeps every directory (AbstractTest.java:143).
    Keeping the launch dir and work dir is needed, because <launchDir>/.nextflow holds the resume cache. But output/ and meta/ are kept too. Stale files in output/ survive, so exists()-style assertions can pass on old data, and meta/ collects nextflow.log.1 rollovers. Clear output/ and meta/ on each run.

  3. Docs. Add a docs note and say that results are not hermetic.

Tests to add

  • Unit test: with -resume in the options, marker files in the launch dir and work dir survive setup(). The current unit test only covers the false case.
  • Integration test: the fixed version of item 3.
  • NXF_HOME is not overridden, with NXF_OFFLINE=true.
  • --resume (double-dash) does not trigger resume, and -resume appears only once on the command line.
  • --update-snapshot combined with -resume is rejected or warns.
  • Stale output/ contents are cleared on resume.
  • AbstractTestDevResumeTest sits in AbstractTestTest.java with unused imports and a placeholder comment.

Nits

  • Unrelated whitespace changes in README.md, docs/installation.md, Config.java and RunTestsCommand.java.
  • In Command.java the environment != null check is dead.
  • buildArgs() and execute() both create the params file.

Not verified

  • Function tests under resume.
  • Shard and --ci interactions.
  • Third-party ITest implementers. Every implementer in the repo extends AbstractTest.
  • Whether editing a test moves it to a new launch dir and loses the cache. I didn't read getHash().

🤖 Generated with Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants