Skip to content

fix(hub): retry each helper notice until its coordinator can take it - #43

Closed
TheGreatAxios wants to merge 2 commits into
mainfrom
cl-11895-stop-one-failed-helper-notice-from-blocking-or-dropping
Closed

TheGreatAxios wants to merge 2 commits into
mainfrom
cl-11895-stop-one-failed-helper-notice-from-blocking-or-dropping

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Contributor

When a helper run ends, the hub tells its coordinator. Before this change:

  • one unreachable coordinator blocked every other project's notice, because each pass retried the oldest unsent notice first;
  • a notice for a project with no active coordinator, or whose coordinator had ended but was still bound, was marked "not notified" and never sent.

Now each ended run carries its own retry state (next_attempt_at, attempts, migration 0036), following Interchange's dispatch-store pattern:

  • mail errors of 500 and above retry on Interchange's dispatch delay (capped at 30 s);
  • no coordinator, an ended coordinator, or a 409 from the mail route waits with backoff (capped at 5 minutes) and delivers once a coordinator can take it;
  • other refusals still settle as not notified;
  • the upsert guard keeps mail-once.

Closes CL-11895

Verification

  • lint, format:check, typecheck, test (70 pass).
  • New e2e test/e2e/specialist-end-notices.test.ts on the throwaway compose DB, asserting stored rows and recipients. 3 tests: a failing coordinator doesn't block another project; an unbound project gets its notice after a bind; a failed but still-bound coordinator's notice goes to the next one. main: 0/3 pass. With this change: 3/3, run 3 times.
  • Independent review, including a concurrency check (two reconcilers): each coordinator mailed exactly once. Its findings (the ended-coordinator case, test independence, retry cap) are fixed.

@corbits-triage

corbits-triage Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Triage: Unsure

Asking a maintainer about:

  • changes sensitive paths: schema: apps/hub/src/db/migrations/0036_ended_run_retry.sql, schema: apps/hub/src/db/migrations/meta/0036_snapshot.json, schema: apps/hub/src/db/migrations/meta/_journal.json

Changed since the last verdict (Fail at bdd4776): 1 new commit.

@corbits-triage corbits-triage Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Triage requests changes:

  • Conventions: added code matches a code pattern

@corbits-triage
corbits-triage Bot dismissed their stale review October 31, 2026 22:08

Nothing triage asked to change still fails, so this request no longer stands.

@TheGreatAxios

Copy link
Copy Markdown
Contributor Author

Verified against a real throwaway Postgres 17: the 3 new e2e tests in test/e2e/specialist-end-notices.test.ts pass, and all 3 fail with specialist-ended.ts reverted to main (schema and migration kept). bun run test: 70 pass.

@TheGreatAxios
TheGreatAxios marked this pull request as draft November 1, 2026 17:52
@TheGreatAxios
TheGreatAxios marked this pull request as ready for review November 1, 2026 17:52
@TheGreatAxios

Copy link
Copy Markdown
Contributor Author

Closing: this rebuilds retry and backoff that Interchange already does natively. Once the hub accepts a mail, Interchange queues it and retries delivery. A coordinator that refuses mail is a fault to surface, not something to retry around. The real bug, one failed notice blocking every other project's, is redone leanly in CL-11918 with no migration and no retry timers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant