Skip to content

bridges: give each concurrent process its own Machine slot - #43

Open
guygrigsby wants to merge 4 commits into
mainfrom
guygrigsby/APT-330/bridge-concurrency
Open

guygrigsby wants to merge 4 commits into
mainfrom
guygrigsby/APT-330/bridge-concurrency

Conversation

@guygrigsby

Copy link
Copy Markdown
Contributor

Parallel aperture sessions in bridge mode evicted each other (APT-330). Every process opened the same state directory for a bridge, so every process registered the same node key, and the control plane handed the session to the newest registrant. The older processes showed Running locally while their dials reached nothing, which surfaced as hangs and "api error". The shared directory was also being written concurrently.

This PR numbers a bridge's Machine identity by slot. Slot 1 keeps the existing state directory and hostname, so devices users already authorized carry over. Slots 2 and up get suffixed siblings: bridges/<hex>-N on disk and aperture-cli-<bridgeID>-N in the admin console. A process claims the lowest free slot when it builds the node and holds it until the Machine closes or is destroyed. The claim is an exclusive non-blocking lock on bridges/locks/<hex>-<slot>.lock, flock on unix and LockFileEx on Windows. Locks live outside the state directory so removal never deletes an open lock.

Machines.Destroy now logs out every slot the bridge has on disk. It refuses, before any logout, when another live process holds a slot and names the conflict. The removal screens list every device the bridge registered, the way the admin console shows them.

TS_AUTHKEY, when set, authorizes a fresh slot without the browser login. Once a slot has registered, its state directory carries the credentials and the key is not consulted again for it. The README gains a section on concurrent sessions and a table of every environment variable the launcher reads. ADR 0006 records the decision and why a shared daemon and ephemeral per-process nodes were rejected.

Risks:

  • Each concurrent process is its own device in the admin console. Ten parallel sessions are ten devices. A process that dies without closing leaves its device listed offline until the bridge is removed.
  • Every new slot needs one browser login unless TS_AUTHKEY is set. Agent fleets need a reusable key or each slot asks once. The key decides which tailnet the fresh device joins, so a key from the wrong tailnet registers the device there.
  • Removing a bridge fails while another aperture process has it open. That case used to log the live session out silently, so this is the intended trade, but it is a new error users will see.
  • A Destroy that returns at its deadline while the node's Close hangs keeps slot 1 locked until the close finishes. A reopen in that window takes slot 2, a fresh device and a fresh login, where it used to wait on the stuck close.
  • Slots are capped at 100 per bridge. Past that the process errors rather than probing further.

APT-330: parallel aperture sessions in bridge mode shared one state directory per bridge, so every process registered the same node key and the control plane handed the session to the newest registrant. The older processes looked Running locally while their dials reached nothing, surfacing as hangs and "api error", and the shared directory was being written concurrently on top.

A Machine's identity is now numbered by slot. Slot 1 keeps the existing state directory and hostname so authorized devices carry over; slots 2 and up get suffixed siblings. A process claims the lowest free slot with an exclusive non-blocking lock on a lock file under bridges/locks/ when it builds the node and holds it until the Machine closes or is destroyed. Lock files live outside the state directory so removal never deletes an open lock and a claimant's inode cannot be deleted under it.

Machines.Destroy logs out every slot the bridge has on disk and fails before any logout when a live process holds one, naming the conflict. Evicting a running session is the silent kill this change removes, not a removal feature.

Rejected the shared daemon (a wire protocol, trust boundary and daemon lifecycle for a first iteration) and ephemeral per-process nodes (key expiry forces a browser login per launch). ADR 0006 records the call and the revisit conditions.
ADR 0006 makes every concurrent process its own device, so a fresh slot needed one browser login each before it could join. With TS_AUTHKEY set, tsnet registers the slot non-interactively; a reusable key covers every slot an agent fleet claims. Once a slot has registered, its state directory carries the credentials and the key is not consulted again for it.
APT-330 taught users the hard way that concurrent sessions evict each
other; the fix (ADR 0006) and the TS_AUTHKEY escape hatch for unattended
sessions were only discoverable by reading the code.
The flags table only names the two vars that mirror flags; TS_AUTHKEY,
the Codex install vars and the incidental reads were undocumented.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant