Skip to content

fix(recovery): wait for concurrent disposable container removal - #882

Open
gael-obin wants to merge 1 commit into
codefly-dev:mainfrom
gael-obin:fix/disposable-removal-race-20261103
Open

gael-obin wants to merge 1 commit into
codefly-dev:mainfrom
gael-obin:fix/disposable-removal-race-20261103

Conversation

@gael-obin

Copy link
Copy Markdown
Contributor

Disposable recovery can select a container while another cleanup operation is already removing it. Docker returns a specific Conflict, which previously aborted service initialization even when that cleanup was succeeding.

The existing Core recovery path now retries only the typed removal-in-progress conflict for the same immutable container ID, within the existing ten-second per-container deadline. Success requires a successful removal or NotFound. Cancellation, unrelated conflicts and other errors still propagate. Eligibility, namespace/live-owner checks, stateful/ledger retention, and volume handling are unchanged.

Classification: fix at the generic container recovery owner. It reuses the current Docker client, scope selection and lifecycle rather than adding a consumer cleanup path or weakening startup checks. This is separate from source watcher recovery.

Validation: a real-Docker concurrent-removal regression on isolated test-owned containers reproduced the old error and passes with the fix under the race detector. Pre-cancellation leaves the container intact; pure classifier tests reject wrong IDs, unrelated conflicts and untyped messages. Existing scope/live-owner predicate checks pass. No mocked Docker client, broad recovery sweep, existing service restart, volume deletion or release was used for this proof.

Limits: conflict timing is scheduler-dependent; a single passing concurrent run does not establish that every possible schedule was exercised. Cancellation/deadline while a real removal remains persistently in progress has not been induced. Exact diagnostic matching follows the supported Docker engine behavior and deliberately surfaces other daemon errors. CLI adoption and downstream integration remain required after owner publication; this source PR does not claim a released consumer is repaired.

Independent final architecture and elegance reviews found no blocking defect in b87cd2b2f7524625169f823bc962d55f57054487. The implementation reviewer separately reproduced the original failure with a local Go overlay and passed the fixed real-Docker suite three times under the race detector. No existing runtime or canonical checkout was changed. Hosted CI is pending on this new PR.

Publish the previously exercised owner implementation for draft review.
@gael-obin

Copy link
Copy Markdown
Contributor Author

@antoinetoussaint-byte Please review and merge this bounded recovery fix at b87cd2b2f7524625169f823bc962d55f57054487 when satisfied. All eight applicable hosted checks now pass (two other checks skipped). Independent implementation review reproduced the original real-Docker concurrent-removal failure, then passed the fixed suite three times under the race detector. The change preserves resource eligibility and the existing deadline; its remaining timing-test limits are in the PR description.

The separate watcher recovery fix at c1a62d1b555fc416bb0974b3900e38695b9103d5 also has eight passing checks. Please review it for the same owner merge batch; its CLI consumer still needs subsequent dependency adoption. These changes do not depend on accepting the separate installation-setup contract.

This account has read-only upstream permissions and GitHub refused a formal reviewer request, so this comment is the approval handoff. No merge, queue entry, tag, release or deployment is claimed. After accepted merges, the normal Core release/consumer adoption path is needed; source merge alone does not repair an already released CLI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant