Skip to content

fix(run): hop route refusals name both addresses; the first-Init port guard probes the proposed ports (r129, r141) - #1180

Merged
antoinetoussaint-byte merged 3 commits into
mainfrom
fix/hop-route-refusals-name-addresses
Oct 26, 2026
Merged

antoinetoussaint-byte merged 3 commits into
mainfrom
fix/hop-route-refusals-name-addresses

Conversation

@antoinetoussaint-byte

@antoinetoussaint-byte antoinetoussaint-byte commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #1178, closing the minors from review r129 (no blockers or majors).

f1/f4: refusals name both addresses and give the right remedy

Take a routed module that is ordered after the hop and whose Init accepts a different address from the one the snapshot froze. It was told "stop and restart", which never helps. The frozen address is the one its Init was proposed, and that address is derived from the workspace, the service and the naming scope, so a restart derives it again and is refused again. The refusal now reads:

hop route platform/second/web::http is routed at [container http://host.docker.internal:13001, native http://localhost:13001], the address platform/second was proposed, but its Init accepted [native http://moved.native]: the proposed address is derived from the workspace, the service and the naming scope, so restarting this run derives it again and is refused the same way; have platform/second's agent serve the proposed address, or order platform/second before the hop

The three real restart cases keep the restart boundary, and each now prints both addresses:

  • an initialized module offers a different address on reload;
  • a record lands between the hop's derivation and its freeze ("was resolved at …, but … initialized at … while the routes were being resolved");
  • a route moved ("moved from X, the address this run's hop snapshot holds, to Y").

The garbled "is not the address … initialized with" wording is gone.

f2: the fence's three guards are pinned

New tests:

  • TestHopFreezeRefusesARecordBetweenDerivationAndFreeze
  • TestHopFrozenRouteRefusesADuplicateEndpointAtAnotherAddress (covers both orders, plus carriesMapping directly)
  • TestHopFreezeRefusesARouteThatMovedBeforeItsOwnerInitialized

Each one was sabotage-checked:

Sabotage Failing test
S2: derive-then-record check in the freeze forced off …RecordBetweenDerivationAndFreeze
S3: carriesMapping changed to any-match …DuplicateEndpointAtAnotherAddress
S3': carriesMapping changed to first-match …DuplicateEndpointAtAnotherAddress
S4: moved compare disabled …MovedBeforeItsOwnerInitialized
S1: first-record check skipped (from #1178) …InitializesAfterTheHop and …DuplicateEndpoint…

f3 / r141-f1: the first-Init port guard actually probes, and the docs say what it guarantees

r141 found that checkInitialPortAvailability probed runner.networkMappings. That field is assigned only after the agent's Init is accepted, so on a first Init the guard probed nothing, and it has done nothing there since cbcbafc. The guard now:

  • runs after the Init inputs are gathered;
  • probes the mappings this Init proposes, before the agent's Init;
  • dials each native port on the instance's own host and on 127.0.0.1 and ::1 explicitly. On macOS, a squatter on 127.0.0.1:P and a service listening on the wildcard :P both bind, so only probing both families before the service binds catches either squatter.

Tests:

  • TestFirstInitRefusesAProposedPortAnotherProcessHolds builds the Runner the way a run does (NewRunner, a real agent process) and holds the proposed port on each family.
  • TestBoundNativePortsProbesBothLoopbackFamilies holds the family the instance does not name.

Sabotage: probing the accepted set fails the first test, and dialling only the instance's host fails the second.

docs/orchestration.md and the freezeHopAddresses comment now say exactly what holds. The routed module's first Init refuses a held proposed port and names its holder, so the run never comes up with the hop routing to a squatter. Not covered:

  • traffic the hop forwarded before that Init;
  • a process that binds the port after the probe;
  • a routed module that crashes or is withdrawn mid-run.

r150-b1: the service's own kept container is not a squatter

A Docker store with keep-running leaves its container up so that the next run's agent adopts it by name. That container keeps publishing the service's deterministic port, and the guard refused it ("held by OrbStack Helper") on every run after the first. The remedy it printed, codefly clear, would have deleted the very container the setting keeps.

The guard now attributes each listener before refusing it. An address is let through only when the container listening there is the one the agent will adopt. It must:

  • carry the name every Core agent gives it (dockerrun.ContainerName over the service's unique name in the run's workspace and naming scope);
  • be codefly-owned;
  • carry this flow's exact recovery scope and, when it has one, the same namespace (dockerrun's own reuse rule);
  • be running and not ephemeral;
  • publish that port on that loopback family: 127.0.0.1/0.0.0.0 covers 127.0.0.1, ::1/:: covers ::1.

Without a recovery identity or a Docker engine, nothing is excused. A loopback instance host is now probed per family only, so every holder is attributed to one family. A refusal names the holder on the address that answered, so a squatter on ::1 beside a kept 127.0.0.1 container is named, not Docker's forwarder.

Tests: TestFirstInitAdoptsItsOwnKeptContainerAndRefusesAnyOtherHolder uses a real Runner from NewRunner with a real agent. The kept container is adopted. Each of these is refused:

  • a holder on ::1 beside it;
  • a container from another recovery scope;
  • one from another namespace;
  • one with another service's name;
  • an ephemeral container;
  • a stopped container;
  • one published on another port.

TestPublishesAddressIsPerLoopbackFamily and TestHolderPIDNamesTheListenerOnTheAddressThatAnswered pin the family and attribution rules. Each of these sabotages turns a case red: no excuse, any family, no scope check, no name check, the identity's (empty) workspace instead of the run's, and dialling a loopback host directly.

Live, on OrbStack with postgres agent 0.0.144, keep-running, container runtime, scope hfkr:

  • Run 1 comes up and leaves its container on 127.0.0.1:35830.
  • Run 2 with e0a9aa0 is refused, naming OrbStack Helper.
  • Run 2 with this change reattaches in 1.2s.
  • With a squatter on [::1]:35830 beside the kept container, the run is refused, naming the squatter's pid and executable.

r150-m1: the stale test comment now says what the test covers, a re-Init before the first Start.

r141-f2: a record that lands between the hop's derivation and its freeze, when it is the module's first record, now gets the non-restart refusal. To tell the cases apart, the shared state counts records per service. Only a re-record inside the window keeps the restart boundary.

r141-f3: "order X before the hop" is now offered only "if X does not call through the hop".

r141-f4: the reload refusal ("now offers") is asserted with both address lists.

Checks

go test -race ./pkg/orchestration/ ./cmd/run/, go vet and golangci-lint run ./pkg/orchestration/ are clean. Classification: a fix, made at the places that own the fence and the port guard. About 110 hand-written lines plus tests.

Behaviour change worth knowing: now that the first-Init guard probes, a stale process from an earlier run in the same naming scope that still holds a deterministic port fails that service's first Init, naming the holder. That was the guard's stated purpose; before this change it never fired on a first Init.

🤖 Generated with Claude Code

antoinetoussaint-byte and others added 2 commits October 25, 2026 23:32
…ee guards (r129)

Follow-up to #1178's review r129 (all minor).

f1/f4: a routed module ordered after the hop whose Init accepts another
address than the one the snapshot froze was told "stop and restart", which
can never help: the frozen address is the one its Init was proposed,
derived from the workspace, the service and the naming scope, so a restart
derives it again. That refusal now names the route, the address the
snapshot holds and the one offered, says a restart is refused the same way,
and gives the remedies (the agent serves the proposed address, or the
module is ordered before the hop). The other refusals — an initialized
module offering another address on reload, a record landing between the
hop's derivation and its freeze, and a route that moved — keep the restart
boundary and now print both addresses; the garbled "is not the address …
initialized with" is gone.

f2: tests pin the three guards no test covered. Each fails under its
sabotage: the derive-then-record check in the freeze forced off, carriesMapping
back to any-match or first-match, and the moved compare disabled.

f3: the window in which the hop routes to a derived loopback port nobody
holds yet is documented, with its bound (the module's first Init finds the
port held and fails the run naming the holder). The hop does not refuse a
held port itself: a routed module whose Init starts its server before
recording holds its own port while that Init is in flight, and would be
refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… loopbacks (r141)

r141-f1 (blocker): checkInitialPortAvailability probed runner.networkMappings,
which is assigned only after the agent's Init is accepted, so on a first Init
it probed nothing — the guard has been vacuous there since cbcbafc, and the
squatting bound #1180 documented rested on it. It now runs after the Init
inputs are gathered and probes the mappings this Init proposes, before the
agent's Init. Each native port is dialled on the instance's own host and on
127.0.0.1 and ::1 explicitly: the two families are separate sockets, and on
macOS a squatter on 127.0.0.1:P coexists with a service's later wildcard bind
on :P. The docs and the freezeHopAddresses comment now say exactly what holds:
the routed module's first Init refuses a held proposed port naming its holder,
so the run never comes up routing to a squatter; traffic forwarded before that
Init, a bind after the probe, and a crashed or withdrawn module are not covered.

r141-f2: a record that lands between the hop's derivation and its freeze is
that module's first record when it has recorded once, so it gets the same
non-restart refusal as any first Init that accepted another address; only a
re-record inside the window keeps the restart boundary.

r141-f3: "order X before the hop" is offered only to a module that does not
call through the hop.

r141-f4: the reload refusal's two addresses are asserted in full.

Tests: TestFirstInitRefusesAProposedPortAnotherProcessHolds builds the Runner
as a run does (NewRunner, a real agent) and holds the proposed port on
127.0.0.1 and on ::1; TestBoundNativePortsProbesBothLoopbackFamilies holds the
family the instance does not name. Sabotage: probing the accepted set fails
the first; dialling only the instance's host fails the second; the f2, f3 and
f4 assertions each fail when their change is reverted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@antoinetoussaint-byte antoinetoussaint-byte changed the title fix(run): hop route refusals name both addresses; pin the fence's three guards (r129) fix(run): hop route refusals name both addresses; the first-Init port guard probes the proposed ports (r129, r141) Oct 26, 2026
…pt container (r150)

r150-b1 (blocker): a Docker store with keep-running leaves its container up
so the next run's agent adopts it by name, and that container still publishes
the service's deterministic port — the one the next Init proposes. The guard
refused it as a squatter ("held by OrbStack Helper") on every run after the
first, and its remedy, 'codefly clear', would delete the container the
setting keeps.

The guard now attributes a listener before refusing it. An address is let
through only when the container listening there is the one the agent will
adopt: its name as every Core agent spells it (dockerrun.ContainerName over
the service's unique name in the run's workspace and naming scope — the
workspace RuntimeInstance.Load sends, not the identity's, which is empty in a
run), codefly-owned, labelled with this flow's exact recovery scope and, when
it carries one, its namespace (dockerrun's own reuse rule), running, not
ephemeral, and publishing that port on that loopback family (127.0.0.1 or
0.0.0.0 for 127.0.0.1; ::1 or :: for ::1). The scope comes from the identity
the flow projected (Runner.containerRecoveryIdentity); with none, or no Docker
engine, nothing is excused. A loopback instance host is now probed per family
only, so every holder is attributed to one family, and the holder a refusal
names is the one on the address that answered — a ::1 squatter beside a kept
127.0.0.1 container is refused naming the squatter, not Docker's forwarder.

r150-m1: TestInitRejectsAStalePortBeforeTheFirstStart's comment says what it
covers now: a re-Init before the first Start, whose proposal is the pinned
accepted set.

Tests: TestFirstInitAdoptsItsOwnKeptContainerAndRefusesAnyOtherHolder (a real
Runner from NewRunner with a real agent): the kept container is adopted; a
::1 holder beside it, another recovery scope, another namespace, another
service's name, an ephemeral container, a stopped one and one published on
another port are each refused. TestPublishesAddressIsPerLoopbackFamily and
TestHolderPIDNamesTheListenerOnTheAddressThatAnswered pin the family and
attribution rules. Sabotage: no excuse, any family, no scope check, no name
check, the identity's workspace instead of the run's, and dialling a
loopback host directly each turn a case red.

Live (OrbStack, postgres agent 0.0.144, keep-running, container runtime,
scope hfkr): run 1 comes up and leaves codefly-hfkeep-keeprun-store-hfkr on
127.0.0.1:35830; run 2 with e0a9aa0 is refused naming OrbStack Helper; run 2
with this change reattaches in 1.2s; with a squatter on [::1]:35830 beside the
kept container the run is refused naming the squatter's pid and executable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@antoinetoussaint-byte
antoinetoussaint-byte added this pull request to the merge queue Oct 26, 2026
Merged via the queue into main with commit 931b09e Oct 26, 2026
15 checks passed
crisfiorini-oss pushed a commit to crisfiorini-oss/cli that referenced this pull request Oct 28, 2026
…t kept (codefly-dev#1201)

* fix(run): the first-Init port guard excuses every container this service's agent kept, not just one named after the service

An agent may run SEVERAL containers for one service and names each after
itself: service-object-storage publishes the service's own grpc port from
`<unique>-gateway` and keeps `<unique>-minio` and `<unique>-network` beside it.
The guard added in codefly-dev#1180 excused only a container named exactly
`ContainerName(<unique>)`, so it excused none of them — and `adoptableContainer`
could therefore never match a keep-running agent that suffixes its container
names, which is the fleet's actual convention.

What that cost: the first flow of a job starts the service fine and leaves its
gateway container up, publishing the service's deterministic port. A later flow
in the same job builds a fresh Runner, so `everStarted` is false and the guard
runs, finds the port bound by Docker's forwarder, fails to attribute it to this
service's own container, and refuses. Deterministically, in every phase, which
is what saas#1265 hit on both of its runs — the first run of that graph on a
release carrying codefly-dev#1180.

A container is now this service's when its logical name label and its Docker
name are the service's name or that name with a suffix. Every other guard is
unchanged: the codefly owner label, this flow's exact recovery scope, the
namespace rule, running, non-ephemeral, and the container must publish the
proposed port on that address's own loopback family. A sibling service is still
not this one, and a squatter is still refused on either family.

Diagnosed by reproduction rather than inference, with the RELEASED 0.1.219 in a
repo unrelated to saas: `codefly run service documents/object-storage
--init-only` twice in one workspace and scope leaves
`…-object-storage-osproof-gateway` publishing 0.0.0.0:47103 and then refuses
`port 47103 (grpc) held by pid … (OrbStack Helper)` — the Docker forwarder, so
the holder is the kept container and neither a leaked process nor a port
derivation. The same second flow passes with this fix. Restoring the old
comparison makes the two new unit cases fail with that same refusal.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(orchestration): require exact parent attribution for sidecars

* docs(orchestration): bound container attribution claims

* fix(conformance): track the selected Core release line

---------

Co-authored-by: Antoine Toussaint <312536060+antoinetoussaint-byte@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
crisfiorini-oss pushed a commit to crisfiorini-oss/cli that referenced this pull request Oct 28, 2026
…t kept (codefly-dev#1201)

* fix(run): the first-Init port guard excuses every container this service's agent kept, not just one named after the service

An agent may run SEVERAL containers for one service and names each after
itself: service-object-storage publishes the service's own grpc port from
`<unique>-gateway` and keeps `<unique>-minio` and `<unique>-network` beside it.
The guard added in codefly-dev#1180 excused only a container named exactly
`ContainerName(<unique>)`, so it excused none of them — and `adoptableContainer`
could therefore never match a keep-running agent that suffixes its container
names, which is the fleet's actual convention.

What that cost: the first flow of a job starts the service fine and leaves its
gateway container up, publishing the service's deterministic port. A later flow
in the same job builds a fresh Runner, so `everStarted` is false and the guard
runs, finds the port bound by Docker's forwarder, fails to attribute it to this
service's own container, and refuses. Deterministically, in every phase, which
is what saas#1265 hit on both of its runs — the first run of that graph on a
release carrying codefly-dev#1180.

A container is now this service's when its logical name label and its Docker
name are the service's name or that name with a suffix. Every other guard is
unchanged: the codefly owner label, this flow's exact recovery scope, the
namespace rule, running, non-ephemeral, and the container must publish the
proposed port on that address's own loopback family. A sibling service is still
not this one, and a squatter is still refused on either family.

Diagnosed by reproduction rather than inference, with the RELEASED 0.1.219 in a
repo unrelated to saas: `codefly run service documents/object-storage
--init-only` twice in one workspace and scope leaves
`…-object-storage-osproof-gateway` publishing 0.0.0.0:47103 and then refuses
`port 47103 (grpc) held by pid … (OrbStack Helper)` — the Docker forwarder, so
the holder is the kept container and neither a leaked process nor a port
derivation. The same second flow passes with this fix. Restoring the old
comparison makes the two new unit cases fail with that same refusal.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(orchestration): require exact parent attribution for sidecars

* docs(orchestration): bound container attribution claims

* fix(conformance): track the selected Core release line

---------

Co-authored-by: Antoine Toussaint <312536060+antoinetoussaint-byte@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant