fix(runtime,modal): retry a continuing task behind a failed consumer preparation, and do not resume over skipped tasks (#1231)

[aviggiano]

Follow-up to #1254 (issue #1231).

Problem

1. resume --retry-failed left a failed continuing producer failed for good when its consumer had failed in preparation. fix(runtime): resume --retry-failed leaves a continuing task its started consumers ran without (#1231) #1254 counted a consumer as started when any of its prepare:, node: or verify: nodes was in-progress, finished, failed or stalled. A consumer whose prepare: node failed or stalled has not admitted its optional inputs yet. Its preparation depends on the producers' verifiers, so on retry it reruns behind the producer's rerun and can read it. Example: a failed lens and a failed prepare:catalog. Resume reset prepare:catalog but not the lens. The catalog then ran without the lens, the lens stayed failed, and WORKFLOW_RETRY_SKIPPED wrongly said the catalog "already ran without it". Before fix(runtime): resume --retry-failed leaves a continuing task its started consumers ran without (#1231) #1254, both were reset.
2. Modal started a resume that resets nothing. modalDurableRunNeedsResume counted skipped checkpoint entries as failures a resume could rerun, but readRetainedFailureStateIds never retains a skipped entry. On the default topology, dedupe-findings lists stateful-invariant-setup as an optional input. When the setup fails, it is retained and its later stages in the specialists group end skipped. The worker then resumed the run, --retry-failed reset nothing, and waitForTerminalRun waited until EVAL_WATCH_TIMEOUT_SECONDS and ended unreachable.

Change

• failedProducersStartedConsumersOmittedInWorkflow (packages/runtime/src/retry-failed-omissions.ts) now counts a consumer as started only when its prepare: node is in-progress or finished.
  • A failed or stalled preparation no longer holds the producer back, so both are retried.
  • A consumer whose agent or verifier failed after its preparation finished still counts as started. node:X depends only on prepare:X, so its rerun starts at once and is admitted before the producer's rerun verifies. Retrying the producer in that case would bring back the false COMPLETE that resume --retry-failed can report COMPLETE after rerunning a continuing node its consumers already ran without #1231 fixed. This is why the PR does not simply skip every failed consumer.
  • An in-progress preparation still counts as started. That errs toward leaving the producer failed (run stays PARTIAL) rather than a possible false COMPLETE.
• Modal's reader (readRetainedFailureStateIds) is unchanged. Run state cannot tell a failed preparation from a failed agent, so it still counts a failed consumer as started. That means Modal may name more retained failures than resume does, which can only skip a resume, never start one that resets nothing. A comment now says so.
• modalDurableRunNeedsResume (packages/modal/src/resume.ts) ignores skipped entries. --retry-failed reruns a skipped task only by rerunning the failed task it waited on, and that task has its own failed entry. If a run has only skipped entries and no failed ones, it still resumes as before.
• Amends the resume --retry-failed can report COMPLETE after rerunning a continuing node its consumers already ran without #1231 CHANGELOG entry and the resume --retry-failed paragraph in docs/reference/cli.md.

Not changed: runSmithersLifecycleCommand still reads input.tasks ?? []. The only caller that passes retryFailed also passes tasks.

Tests

• New runtime tests (runtime.test.ts), each with a failed lens and a different catalog state:
  • prepare:catalog failed or stalled: resume resets both node:lens and prepare:catalog, and reports no WORKFLOW_RETRY_SKIPPED. Both cases fail without the fix.
  • node:catalog failed, or verify:catalog failed: resume resets the catalog only, leaves the lens failed, and reports WORKFLOW_RETRY_SKIPPED.
• New Modal test (resume.test.ts): a failed, retained stateful-invariant-setup with skipped later stages does not resume. It still resumes without the retained set, or when another task failed. The test fails without the fix.
• Existing runtime tests for --retry-failed, readRetainedFailureStateIds and the optional-prerequisite sync cases: 21 of 21 pass. Full Modal vitest suite: 524 of 524 pass.
• lint:strict:ci, eslint, prettier, docs:check, and the runtime and modal typechecks are clean.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge, though two provider-home preflight gaps leave avoidable late workflow failures.

Summary

The PR adjusts failed-task retry decisions and Modal resume handling, and its merge incorporates upstream provider-home preflight, OpenRouter transport settings, and an advisory exception.

• Failed consumer preparation can be retried alongside its continuing producer; skipped Modal tasks no longer independently trigger a resume.
• The new provider-home preflight misses OpenRouter and environments without HOME, leaving those late failures without its early diagnostic.

_{Reviews (2) · Last reviewed commit: "Merge branch 'unstable' into fix/1231-re..."}