From 2ae4a0547be3ca316b194b81c9e602a1eca9c6eb Mon Sep 17 00:00:00 2001 From: Daniel Hanold Date: Tue, 20 Oct 2026 06:47:13 -0400 Subject: [PATCH 1/2] Add design spec: Fix the flaky run-cancel worktree test --- ...nd-failing-tests-standing-bucket-design.md | 103 ++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100644 docs/superpowers/specs/2026-10-20-flaky-and-failing-tests-standing-bucket-design.md diff --git a/docs/superpowers/specs/2026-10-20-flaky-and-failing-tests-standing-bucket-design.md b/docs/superpowers/specs/2026-10-20-flaky-and-failing-tests-standing-bucket-design.md new file mode 100644 index 000000000..d9147d1e7 --- /dev/null +++ b/docs/superpowers/specs/2026-10-20-flaky-and-failing-tests-standing-bucket-design.md @@ -0,0 +1,103 @@ +# The run-cancel worktree test waits for the launch to finish before it cancels — design + +Change 0569 — Fix the flaky run-cancel worktree test + +Change #569, groomed 2026-10-20. Found during #562's build gate. + +## The problem + +`TestIntegrationRunCancelFreesWorktreeForNextStart` (`internal/app/runtracker_cancel_integration_test.go`) +failed once in #562's full parallel suite and then passed 3 of 3 alone. The failure was the first +assert: `runCancel` returned `cancellation-pending`, with a `claim-busy:` finding, where the +test wants `cancelled`. + +The test does this today: + +1. `startRunDrive` runs the build drive's `Start` on a goroutine, with the suite command `sleep 60`. +2. `awaitRunPhase(t, runRoot, "running")` polls the run's `manifest.json` until its phase reads + `running`. +3. It checks the supervisor holds the worktree lock, then calls `runCancel` once and expects + `cancelled` plus `replacement-stopped:`. +4. It waits (bounded) for the worktree lock to come free. + +**Cause (traced in code).** Starting a drive takes the drive's launch claim (the claimant flock on +the drive's `relaunch.lock`) in `revalidateAdmittedLaunch` and holds it across the launch. In +`gatedrive.launchScopeless` the order is: + +1. `d.proc.Launch(...)`. The supervisor writes `manifest.json` with phase `running`, then sends the + `running` handshake, and `Launch` returns when it reads the handshake. +2. `d.store.attachLaunch(...)`, an fsync'd record write, then the holder note. +3. `claim.close()`. The claim is released here, before the drive's observation slice. + +The test polls the manifest file directly, so it can see `running` at step 1, before the launcher +reaches step 3. A cancel that lands in that window finds the claim busy (`reconcileRunDrive` returns +`claim-busy:`), and `runCancel` correctly reports `cancellation-pending`. Under the parallel +suite, fsync contention stretches steps 1–2, so the window grows. That matches the failure showing up +only in a full run. + +**The product is right; the test's precondition is too weak.** ADR-0132 names `claim-busy` as one of +the causes of `cancellation-pending`, and the documented remedy is to re-run the cancel. The test +means to say "a run whose drive is running and attached cancels in one call". Its wait only proves +"the suite process has started". + +## The fix + +Test-only. No production code changes. + +After `awaitRunPhase(..., "running")` and before `runCancel`, the test also waits until the launch +is attached and its claim is released. The wait uses the same bounded-deadline, short-poll shape as +the test's other waits, never a fixed sleep. + +**Suggested signal (builder may pick another one that is just as deterministic).** The production +read-side census, `productionCancelSeams(fx.repo).launchObserver.observe(fx.contextHash)` (it wraps +`gatedrive.Driver.VerdictRunLaunches`). While the launch is in flight it reports +`claim-busy:`. Once the claim is released (the run dir is attached by then) it reports +`run-live:`. Wait until the findings contain `run-live:` and not `claim-busy:`. + +- This census stops and signals nothing. The only thing it ever settles is a *proven + never-launched* first launch, and it reaches that path only while the claim is free and no run dir + is attached. Once the manifest says `running`, the launcher already holds the claim (or has + attached and released it), so that path can't be reached here. Keep the order: poll only *after* + `awaitRunPhase(..., "running")`. +- Another acceptable signal is probing the drive's claim lock directly. Either way, the signal must + mean "the launcher has released its claim". "The run dir is recorded" is not enough, because + `attachLaunch` happens before `claim.close()`, so that wait still leaves a gap. + +**What the test keeps asserting, unchanged:** one `runCancel` call; disposition `cancelled`; the +`replacement-stopped:` finding; then the bounded wait for the worktree lock to come free. + +**Not allowed:** retrying the cancel, accepting `cancellation-pending`, lengthening a timeout as the +fix, or removing or loosening any existing assert. + +## Proving the fix + +1. **Make the failure happen on demand first.** Temporarily (never committed) add a delay of a few + hundred milliseconds in `launchScopeless` between `attachLaunch` and `claim.close()`. The + *unchanged* test must fail with `cancellation-pending` / `claim-busy`. +2. **Same delay, fixed test:** it passes. +3. Remove the delay. Confirm with `git diff` that no production file changed. +4. Run the fixed test alone, repeated (for example `-count=50`), and run the full suite once at the + build gate. + +Record steps 1–3 (the delay used, the red message, the green re-run) in the results file. + +## Sweep for the same shape + +Look for other tests that cancel a run, or count on the census being settled, right after seeing a +run reach `running` while its `Start` goroutine may still be mid-launch. Find them with a +repo-wide grep (`awaitRunPhase`, `runCancel`, `ReconcileRunLaunches`, `VerdictRunLaunches`, +`"running"` waits in `_test.go` files), not by listing this file alone. Fix any that turn up the same +way, or say in the results that there were none. + +A sweep at grooming time found none. The only other `awaitRunPhase(..., "running")` caller, +`TestIntegrationRunCancelSignaledOrVanishedSupervisorIsCancelled` (its `vanished` case), waits for +`Start` to return (`awaitStart`) before it cancels, and `Start` returns only after `claim.close()`. +The other census and cancel tests use seeded drive records or synchronous starts, or interleave +with a launch on purpose using fakes (`driver_concurrency_test.go`, `driver_launch_claim_test.go`). +Re-run the grep at build time in case anything new landed. + +## Out of scope + +- Changing how cancel, the census, or the launch claim behave (ADR-0132). +- Any other flaky test. A later one gets its own change. +- Slow tests and budget lines (the budget report owns those). From b5e6d4883fb2370c460445d57f762c49a357a57f Mon Sep 17 00:00:00 2001 From: Daniel Hanold Date: Tue, 20 Oct 2026 06:53:41 -0400 Subject: [PATCH 2/2] test(app): run-cancel worktree test waits for the launch claim to be released before cancelling --- .../app/runtracker_cancel_integration_test.go | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/internal/app/runtracker_cancel_integration_test.go b/internal/app/runtracker_cancel_integration_test.go index 2cfae846e..58c0d4e8e 100644 --- a/internal/app/runtracker_cancel_integration_test.go +++ b/internal/app/runtracker_cancel_integration_test.go @@ -1065,6 +1065,29 @@ func awaitRunPhase(t *testing.T, runRoot string, want ...string) (string, runMan } } +// awaitLaunchSettled waits until the drive id has released its launch claim. The +// run's manifest reaches "running" inside d.proc.Launch, but launchScopeless keeps +// holding the drive's launch claim through attachLaunch until claim.close, and a +// cancel that lands in that gap correctly reports cancellation-pending (ADR-0132). +// The production stop-free launch census reports claim-busy while the claim is +// held and run-live once it is released; it stops nothing. Call it only after +// awaitRunPhase has seen "running". +func awaitLaunchSettled(t *testing.T, fx cancelFixture, id string) { + t.Helper() + observer := productionCancelSeams(fx.repo).launchObserver + deadline := time.Now().Add(30 * time.Second) + for { + report, err := observer.observe(fx.contextHash) + if err == nil && hasFinding(report.Findings, "run-live:"+id) && !hasFinding(report.Findings, "claim-busy:"+id) { + return + } + if time.Now().After(deadline) { + t.Fatalf("drive %s never released its launch claim within 30s (last findings=%v, err=%v)", id, report.Findings, err) + } + time.Sleep(10 * time.Millisecond) + } +} + // awaitStart receives the backgrounded Start's result, failing on a wedge. func awaitStart(t *testing.T, done <-chan GateDriveResult) GateDriveResult { t.Helper() @@ -1106,6 +1129,7 @@ func TestIntegrationRunCancelFreesWorktreeForNextStart(t *testing.T) { t.Fatal("precondition: the running drive's supervisor must hold the worktree lock") } id := onlyDriveID(t, fx) + awaitLaunchSettled(t, fx, id) res := runCancel(productionCancelSeams(fx.repo), fx.repo, fx.key, "human stop") if res.Disposition != CancelDispositionCancelled {