bridges: give each concurrent process its own Machine slot - #43
Open
guygrigsby wants to merge 4 commits into
Open
guygrigsby wants to merge 4 commits into
guygrigsby wants to merge 4 commits into
Conversation
APT-330: parallel aperture sessions in bridge mode shared one state directory per bridge, so every process registered the same node key and the control plane handed the session to the newest registrant. The older processes looked Running locally while their dials reached nothing, surfacing as hangs and "api error", and the shared directory was being written concurrently on top. A Machine's identity is now numbered by slot. Slot 1 keeps the existing state directory and hostname so authorized devices carry over; slots 2 and up get suffixed siblings. A process claims the lowest free slot with an exclusive non-blocking lock on a lock file under bridges/locks/ when it builds the node and holds it until the Machine closes or is destroyed. Lock files live outside the state directory so removal never deletes an open lock and a claimant's inode cannot be deleted under it. Machines.Destroy logs out every slot the bridge has on disk and fails before any logout when a live process holds one, naming the conflict. Evicting a running session is the silent kill this change removes, not a removal feature. Rejected the shared daemon (a wire protocol, trust boundary and daemon lifecycle for a first iteration) and ephemeral per-process nodes (key expiry forces a browser login per launch). ADR 0006 records the call and the revisit conditions.
ADR 0006 makes every concurrent process its own device, so a fresh slot needed one browser login each before it could join. With TS_AUTHKEY set, tsnet registers the slot non-interactively; a reusable key covers every slot an agent fleet claims. Once a slot has registered, its state directory carries the credentials and the key is not consulted again for it.
APT-330 taught users the hard way that concurrent sessions evict each other; the fix (ADR 0006) and the TS_AUTHKEY escape hatch for unattended sessions were only discoverable by reading the code.
The flags table only names the two vars that mirror flags; TS_AUTHKEY, the Codex install vars and the incidental reads were undocumented.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parallel aperture sessions in bridge mode evicted each other (APT-330). Every process opened the same state directory for a bridge, so every process registered the same node key, and the control plane handed the session to the newest registrant. The older processes showed Running locally while their dials reached nothing, which surfaced as hangs and "api error". The shared directory was also being written concurrently.
This PR numbers a bridge's Machine identity by slot. Slot 1 keeps the existing state directory and hostname, so devices users already authorized carry over. Slots 2 and up get suffixed siblings:
bridges/<hex>-Non disk andaperture-cli-<bridgeID>-Nin the admin console. A process claims the lowest free slot when it builds the node and holds it until the Machine closes or is destroyed. The claim is an exclusive non-blocking lock onbridges/locks/<hex>-<slot>.lock,flockon unix andLockFileExon Windows. Locks live outside the state directory so removal never deletes an open lock.Machines.Destroynow logs out every slot the bridge has on disk. It refuses, before any logout, when another live process holds a slot and names the conflict. The removal screens list every device the bridge registered, the way the admin console shows them.TS_AUTHKEY, when set, authorizes a fresh slot without the browser login. Once a slot has registered, its state directory carries the credentials and the key is not consulted again for it. The README gains a section on concurrent sessions and a table of every environment variable the launcher reads. ADR 0006 records the decision and why a shared daemon and ephemeral per-process nodes were rejected.Risks:
TS_AUTHKEYis set. Agent fleets need a reusable key or each slot asks once. The key decides which tailnet the fresh device joins, so a key from the wrong tailnet registers the device there.Destroythat returns at its deadline while the node'sClosehangs keeps slot 1 locked until the close finishes. A reopen in that window takes slot 2, a fresh device and a fresh login, where it used to wait on the stuck close.