fix(runtime): resume --retry-failed leaves a continuing task its started consumers ran without (#1231)

[mrthankyou]

Fixes #1231.

Problem

A failed node in a failure_policy: continue group (a property lens since #1198, or an optional strategy) is an optional input to its consumers in other groups, so they run without it. resume --retry-failed reran it anyway. A succeeded consumer is never finalized again, so the rerun could not reach that consumer's output. Once the rerun succeeded, every planned task read as succeeded, the report said COMPLETE, and a --require-complete run could end succeeded.

Confirmed on unstable (a375d3b4): with the new tests and no fix, resume --retry-failed issued timetravel --node-id node:lens (and node:optional-specialist) after the consumer had finished without it.

Fix

Takes the direction the issue proposes:

• packages/runtime/src/retry-failed-omissions.ts: failedProducersStartedConsumersOmitted names each failed producer that a started consumer lists in optionalDependencyArtifactDirs. It has two readers:
  • The resume reader works from Smithers node states. A consumer counts as started once its prepare:, node: or verify: node is in-progress, finished, failed or stalled.
  • The Modal reader works from state.json and smithers/tasks.json.
• runSmithersLifecycleCommand: --retry-failed no longer resets those producers. It still retries every other failed task. resumeRun reports each skipped producer as a WORKFLOW_RETRY_SKIPPED warning that names the consumers that ran without it. The report stays PARTIAL.
• Modal: modalDurableRunNeedsResume takes the retained failures and no longer resumes a terminal run whose only failures are those. Before, waitForTerminalRun would wait for a change that never came.
• Docs: added the new behavior to docs/reference/cli.md and a CHANGELOG entry.

Not changed: the report-completion schema, and --reset-node, which can still rerun such a node.

Tests

• New runtime tests (runtime.test.ts) cover a lens and a strategy. Each runs with its consumer started (producer left failed, other failed task retried, warning reported) and with its consumer not started (producer retried as before).
• New assertion in the existing sync test "syncRun finalizes consumers admitted without an optional prerequisite whose retry is still running": after the first sync, readFailedProducersStartedConsumersOmitted reports lens → [catalog].
• New Modal unit test for modalDurableRunNeedsResume with retained failures.
• Targeted runtime tests: 13 of 13 pass, including the existing --retry-failed tests and the sync tests for optional prerequisites. Modal resume.test.ts: 32 of 32 pass. eslint and prettier are clean on the changed files. I have not run the full runtime suite locally.

🤖 Generated with Claude Code

Confidence Score: 4/5

The PR is not safe to merge until Modal avoids resuming terminal runs whose only non-retained failed checkpoint entries are skipped tasks.

Findings

1.  Skipped tasks trigger needless resume ▶

Fix with agent prompt

### Issue 1
packages/modal/src/resume.ts:74-77
When a terminal run has a retained failed producer and a skipped downstream task, the checkpoint counts the skipped task as failed. This check therefore requests a resume, even though `--retry-failed` cannot reset the skipped task and leaves the producer failed. With nothing to redo, the Modal worker waits for a state change until its watch times out instead of finalizing the partial evaluation.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR keeps failed optional producers failed once consumers have started without them, reports skipped retries, and uses retained-failure state to decide whether Modal should resume a terminal evaluation.

• Adds runtime omission detection and retry filtering, with lens, strategy, reset, and aggregate tests.
• Updates Modal resume decisions and documents the resulting partial-run behavior.
• The Modal decision still mistakes skipped tasks for work that --retry-failed can redo.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Terminal run] --> B[Read retained producer failures]
  B --> C{Other failed checkpoint IDs?}
  C -->|Retryable task| D[Resume and wait for advancement]
  C -->|None| E[Finalize partial evaluation]
  C -->|Skipped task misclassified| D

Loading

_{Reviews (4) · Last reviewed commit: "Merge branch 'unstable' into fix/1231-re..."}

[mrthankyou]

Real-engine reproduction of the scenario this PR fixes, where the report claimed COMPLETE with 0 issues while the strategies that found the bug were never read: #1231 (latest comment).

[greptile-apps]

Skipped tasks trigger needless resume When a terminal run has a retained failed producer and a skipped downstream task, the checkpoint counts the skipped task as failed. This check therefore requests a resume, even though --retry-failed cannot reset the skipped task and leaves the producer failed. With nothing to redo, the Modal worker waits for a state change until its watch times out instead of finalizing the partial evaluation.

Knowledge Base Used: Modal worker integration

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/modal/src/resume.ts
Line: 74-77

Comment:
**Skipped tasks trigger needless resume** When a terminal run has a retained failed producer and a skipped downstream task, the checkpoint counts the skipped task as failed. This check therefore requests a resume, even though `--retry-failed` cannot reset the skipped task and leaves the producer failed. With nothing to redo, the Modal worker waits for a state change until its watch times out instead of finalizing the partial evaluation.

**Knowledge Base Used:** [Modal worker integration](https://app.greptile.com/monad-foudnation/-/custom-context/knowledge-base/monad-developers/ultrafuzz/-/docs/modal-worker-integration.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.