codefly run that fails or is stopped leaves docker containers and a native ClickHouse group holding ports; codefly clear refuses the group (0.1.222)

Environment: CLI v0.1.222 (darwin/arm64), OrbStack, workspace obin-ai/module-robin playground, codefly run service … --env local --headless over 13 services (authority, documents, model, tracing, evaluation, annotations, robin).

What happens: each time a codefly run service … ends without a clean shutdown, it leaves behind:

docker containers that are still running: codefly-robin-playground-documents-object-storage-gateway, …-object-storage-minio, and …-documents-parse;
a native ClickHouse process group (clickhouse-watchdog plus clickhouse server …), from tracing's evidence store.

"Without a clean shutdown" covered two cases:

the run failed during startup (flow not ready … context deadline exceeded, or cannot initialize …: port … held by pid …), or
the codefly run process was terminated from outside.

The next codefly run then fails on the held ports, for example:

cannot initialize documents/object-storage: port 49843 (grpc) held by pid 18438 (OrbStack Helper)
cannot initialize tracing/evidence: port 10310 (tcp) held by pid 95396 (…/clickhouse server …)

codefly clear doesn't recover from it:

codefly clear --dry-run lists the containers, but then fails: inspect registered group 95384: incompatible boot identity for unchanged process start; preserve the record and recover through its owner.
Without --all, it would also remove unrelated exited containers (codefly-test-mod-svc-*).

Reproduced 4 times on 29 Oct 2026. The workaround each time was docker rm -f on the named containers and kill -TERM -<pgid> on the ClickHouse group.

Expected: a run that fails during startup, or is stopped, stops the containers and process groups it started, as #585 and #502 established for native binaries. And codefly clear can reap a group whose boot identity it can't confirm, or says how to recover it.

Related: #585, #502, #377, #288 (all closed).

Findings from reading v0.1.222 (core v0.19.15). This is two independent problems. #1222 addresses neither: it adds --graceful to recover process PID.

A. codefly clear / the post-run reaper stop on one undecidable group

The refusal. checkBootIdentity (core runners/base/pgid.go) refuses a registered group whose leader has the same PID and start time but a different boot_id. Older Core builds recorded Darwin kern.boottime ("sec:usec"), which changes when the clock is set; newer ones use kern.bootsessionuuid. So a nix-run agent built against older Core (likely the ClickHouse one here) writes records that later look undecidable. Treating that as "don't signal" is right.
The problem is what happens next.
- PreviewManagedProcessGroups aborts the whole listing on the first such error (pkg/processgroup/registered.go).
- A real clear records it and then skips the fallback reapers (cmd/clear.go).
- reconcileProcessGroupRegistries returns before the legacy pass (pkg/processgroup/reaper.go).
- So one stale group blocks all cleanup.
Proposed fix (small): make the refusal per-group and non-fatal. Skip that group, name its PGID and the codefly recover process-group <PGID> --preview / --expect recovery, and keep reaping the rest. Optionally, in Core, accept a legacy "sec:usec" boot id that still equals the current kern.boottime when the owner is dead, as recoveryBootMatches already does for recover.

B. Containers survive a failed or stopped run

What a run does at the end. Teardown sends Stop, never Shutdown/remove. Containers are labelled codefly.session=<agent pid>. On SIGTERM an agent stops its runtime only if it was loaded, within 5 s.
Why the containers stay. Once the agent is gone, shouldReapContainer (dockerrun/container_sweep.go) keeps a running dead-owner container by design (reuse); only --cli-server runs mark containers ephemeral.
Fix direction:
- Stamp a per-run owner label.
- Have a failed or interrupted run shut down or sweep its own containers after ClearAgents.
- SIGKILL is still only reachable by the next run's sweep.
This touches the deliberate dead-owner reuse rule, adopted containers and session-ledger exclusions, so it's a design call for this repo's owners.

I'll open a PR for A.

Part A merged in #1236: a group Core can't decide (e.g. an older agent's boot id) is now kept and named, with codefly recover process-group <PGID> --preview, and codefly clear --dry-run / stop still handle every other group. It ships with the next release after v0.1.224. Leaving this open for part B: containers that outlive a failed or stopped run, which needs a decision on the dead-owner reuse rule.

Part B is in #1241: a run that fails removes the containers its own agents created since it started. A run stopped on purpose keeps them for reuse. It was verified live with a playground run that was made to fail by holding a port it needed; the failed run removed its 2 containers.

Navigation Menu

Repository navigation

codefly run that fails or is stopped leaves docker containers and a native ClickHouse group holding ports; codefly clear refuses the group (0.1.222) #1233

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

codefly run that fails or is stopped leaves docker containers and a native ClickHouse group holding ports; codefly clear refuses the group (0.1.222) #1233

Description

Activity

markwang-obin commented on Oct 29, 2026

markwang-obin commented on Oct 29, 2026

markwang-obin commented on Oct 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions