Skip to content

fix(run): a failed run removes the containers its own agents created - #1241

Merged
markwang-obin merged 2 commits into
mainfrom
fix/failed-run-removes-its-containers
Oct 29, 2026
Merged

markwang-obin merged 2 commits into
mainfrom
fix/failed-run-removes-its-containers

Conversation

@markwang-obin

Copy link
Copy Markdown
Contributor

Part B of #1233. Part A, which keeps an undecidable process group without stopping the rest, landed in #1236.

Problem

A codefly run that fails partway leaves its half-started containers running. Containers outliving a run is deliberate: dockerrun.shouldReapContainer keeps a running container whose owner is gone, so the next run can reuse its dev data. But a failed run holds no such data, and its containers keep the ports the next run needs. The next run then refuses those ports.

Change

A failed run now force-removes the containers that its own agents created since it started, within its recovery scope. A run that is stopped on purpose (by a signal) keeps its containers, as before.

A container counts as the run's own only when all of these hold (createdByRun):

  • it is codefly-owned (codefly.owner=true);
  • it is not a ledgered data container (it has no codefly.invocation);
  • its codefly.session is the pid of one of this run's agents, recorded as each agent loads;
  • it was created after the run started. This means a recycled pid on an adopted container doesn't count.

While another run's managed process groups are live in the workspace, nothing is removed. Nothing records whether that run adopted one of these containers.

The cleanup runs only for runtime contexts that already sweep stale containers (shouldSweepStaleContainers). It has a 30 s budget, and its errors are joined onto the run's error.

Classification: a fix at its owner. The leak is in the CLI's run lifecycle, and this is where it is closed.

Reused:

  • listOwnedContainers and the recovery scope from containerRecoveryIdentity;
  • processgroup.PreviewManagedProcessGroups for the live-run check;
  • the agent PIDs the manager already loads;
  • Core's existing container labels.

There is no new label and no Core change.

Tests

pkg/orchestration/run_containers_test.go:

  • TestOnlyContainersThisRunsAgentsCreatedAreItsOwn, a table covering ours, another agent's, created before the run, ledgered, unowned, and a non-pid session;
  • TestAFailedRunRemovesOnlyTheContainersItCreated;
  • TestAFailedRunKeepsItsContainersWhileAnotherRunIsLive.

go test ./pkg/orchestration ./cmd/run passes, and golangci-lint reports 0 issues.

Live run: I started module-robin's playground with this CLI while a port it needs (127.0.0.1:10310) was held, so startup failed partway, at tracing/evidence, with exit 1. The run logged removed 2 containers the failed run started: [codefly-robin-playground-documents-object-storage-gateway codefly-robin-playground-documents-object-storage-minio], and no robin-playground containers remained afterwards.

Not covered

  • A run killed with SIGKILL runs no cleanup.
  • If this run adopted another run's container, only the live-run check guards it.
  • There is no Docker integration test; the engine is faked in the unit tests.

🤖 Generated with Claude Code

A codefly run that fails partway leaves its half-started containers
running, and they hold the ports the next run needs (#1233, part B).
A failed run now force-removes the containers its own agents created
since it started, in its recovery scope. A run stopped on purpose keeps
them, as before, so the next run reuses their dev data.

A container counts as the run's own only when it is codefly-owned,
carries one of the run's agent pids as its session, was created after
the run started, and is not a ledgered data container. While another
run is live in the workspace nothing is removed, since nothing records
that it adopted one of these containers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@markwang-obin
markwang-obin added this pull request to the merge queue Oct 29, 2026
Merged via the queue into main with commit 920c366 Oct 29, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant