Repository navigation
fix(windows): end sandbox processes when the gateway is killed - #5299
Open
ashwind-nvidia wants to merge 1 commit into
Open
ashwind-nvidia wants to merge 1 commit into
ashwind-nvidia wants to merge 1 commit into
Conversation
When the MXC driver runs inside the gateway, a hard-killed gateway left its sandbox processes (wxc-exec, openshell-supervisor and openshell-windows-sandbox) running with no owner. They also kept the gateway's listening socket open, so the port stayed bound and a new gateway failed to start with "Only one usage of each socket address" (os error 10048). There were two causes, so there are two changes: - The gateway now puts itself in a kill-on-close job object before it spawns anything. Every sandbox process joins the job, and Windows ends them all when the gateway process exits for any reason, including a hard kill. If the job cannot be created the gateway logs a warning and continues. - mio creates listening sockets with plain socket(), which leaves the handle inheritable, and child processes are started with handle inheritance enabled, so each sandbox process held a copy of the socket. The gateway, health and metrics listeners now clear the inherit flag right after bind. Either change releases the port on its own; the job object also removes the orphaned processes. Tests: unit tests check that the process is in a kill-on-close job and that a bound gateway listener is not inheritable and still accepts connections. On a native Windows 11 x64 host with three running sandboxes, hard-killing the gateway left 9 processes and an unbindable port in 5 of 5 runs before this change. With it, no process survived, the port could be bound and a new gateway started in 3 of 3 runs on the final build, and in 5 of 5 runs on an earlier build that differed only by two lint-driven edits. The existing MXC end-to-end scenarios give the same results with and without the change. Signed-off-by: ashwind <ashwind@nvidia.com>
|
All contributors have signed the DCO ✍️ ✅ |
Author
|
I have read the DCO document and I hereby sign the DCO. |
8 tasks
4 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
With the MXC driver built into the gateway, hard-killing the gateway left its sandbox processes running with no owner, and they kept the gateway's listening socket open so a new gateway could not bind the port. The gateway now puts itself in a kill-on-close job object so every sandbox process ends with it, and its listening sockets are no longer inheritable, so no child can keep the port bound.
Related Issue
Closes #5298
Changes
openshell_gateway::process_lifetime::kill_children_on_exit(), called first in the gateway'smain. It puts the process in a job object withJOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE; every sandbox process joins it and Windows ends them when the gateway exits for any reason. If the job cannot be created, a warning is printed and the gateway continues. It does nothing on other platforms.disable_handle_inheritance). mio 1.2.0 creates Windows sockets with plainsocket(), which leaves the handle inheritable, and children are started with handle inheritance enabled.windowscrate as a Windows-only dependency ofopenshell-serverandopenshell-gateway(Cargo.lockupdated).Behavior change: sandboxes now end when the gateway process exits, including a normal restart. Before, they were left running without an owner.
Testing
gateway_listener::tests(2) andprocess_lifetime::tests(1) pass on Windows x64; the server tests that ran alongside (4) pass.wxc-exec,openshell-supervisor,openshell-windows-sandbox), the port could not be bound, and a new gateway failed with os error 10048, in 5 of 5 runs. After: no process survived, the port could be bound and a new gateway started in 3 of 3 runs on the final build, and in 5 of 5 runs on an earlier build that differed only by two lint-driven edits.run-mxc-e2e.ps1): 6 of 7 pass with and without the change;host-servicefails in both, so it is not caused by this change. The runner's port probe did not detect the listening gateway on this host, so a scratch copy with a loopback connect probe was used; the repository copy was not modified.cargo fmt --all -- --checkpasses.cargo clippywith the CI flags reports 322 diagnostics (319large future) both with and without this change and none in the changed files.cargo check --lockedandcargo clippy --locked --all-targets -D warningspass foropenshell-gatewayandopenshell-serveron Linux (Rust 1.95.0).Not tested: ARM64, a graceful gateway shutdown, and the full workspace test suite.
Checklist