Repository navigation
fix(compute): take over driver operations abandoned by a stopped gateway - #17
Open
mattj-monad wants to merge 2 commits into
Open
mattj-monad wants to merge 2 commits into
mattj-monad wants to merge 2 commits into
Conversation
A claimed driver operation stays pending until the process that claimed it
records the driver's response. When that gateway process stopped first, as
in a node rotation that interrupted an automatic restart, nothing could ever
settle it: the restart sweep, retries and deletion all refused the sandbox
("previous driver operation is still pending"), and the agent stayed down
until the store rows were deleted by hand. Upstream tracks the same bug as
NVIDIA#4409.
An in-process driver's operation cannot outlive its process, apart from
requests already sent, which the backend commits or rejects within its own
request timeout. So:
- Each claim records its owner, a per-process incarnation
(SandboxProvisioning.driver_operation_owner), and every gateway process
refreshes a presence row every 10 seconds.
- A process that cannot refresh its presence for a minute fences itself and
abandons the operations it owns, before dispatch if already fenced.
- The reconciler takes over a pending operation once its owner's presence is
3 minutes old, or after watching it for 3 minutes when the owner left no
presence (operations claimed before this change). Taking over settles it
with an unknown outcome and rotates its ID, so a late response is ignored.
- An interrupted automatic restart becomes due again, and the restart sweep
starts a fresh runtime generation, stopping what the abandoned operation
left behind first.
Drivers running in their own process keep the original contract.
Signed-off-by: Matt Jane <matt@monad.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014dzrkBzoiGwdAtxRmRxtR1
…r's clock Review findings on the operation takeover: - A heartbeat that hung (a database host gone without a reset) never failed, so the owner never fenced while another replica could take it over. Each heartbeat now times out after 5s, and the fence is driven by time since the last successful heartbeat began. - Presence was judged by comparing the owner's write timestamp with the reconciler's wall clock, so skew could take over a live owner. The sweep now records when each owner's presence row last changed on its own clock. - An older gateway rewriting a sandbox during a rolling upgrade drops the unknown owner field, and the empty-owner delay was as short as any other. Operations with no recorded owner now wait 15 minutes. The takeover delay is 4 minutes, leaving margin over a 75s fence plus in-flight requests. - Takeover applied to unmanaged out-of-process drivers, because only managed ones set driver_process. The runtime now records an in-process driver explicitly. - Graceful shutdown now fences, so in-flight operations settle rather than wait for a takeover. New tests cover a hanging heartbeat, unfencing, shutdown, an owner kept alive by heartbeats, the unowned delay, and a late response after takeover. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014dzrkBzoiGwdAtxRmRxtR1 Signed-off-by: Matt Jane <matt@monad.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On 2026-10-26 a node rotation on
internaltook down both production agents (gtm-bot, oncall-triage) for about 7.5 hours. Each gateway lost its sandbox runtime and started an automatic restart, and then the gateway process itself restarted mid-restart. The restart's claim (driver_operation_pending) is cleared only by the process that made it, so after that:Retaining pending driver operation during gateway recoveryand moved on,sandbox deletefailed withprevious driver operation is still pending; retry and deletion are blocked.The only way out was deleting the sandbox's store rows by hand. Upstream tracks the same bug as graceful shutdown leaves a pending operation undeletable (NVIDIA/OpenShell#4409).
Fix
With an in-process driver (Kubernetes in our gateway), an operation cannot outlive its process. The only exception is requests already sent, which the API server commits or rejects within its own request timeout. So ownership can be proven by a heartbeat:
replica_id/uuid) in the newSandboxProvisioning.driver_operation_owner. A restarted container counts as a new owner even though it keeps its hostname.gateway_process_presencerow every 10s, in single-replica and HA modes.driver_operation_id, so a late response is rejected. An interrupted automatic restart becomes due again, and the restart sweep starts a fresh runtime generation, stopping whatever the abandoned operation left behind first. Retry and deletion work again. Other kinds of operation are not re-driven, but their provisioning deadline still applies.Only drivers built in-process (
ComputeDriverInstance::InProcess) are taken over. Managed and unmanaged remote drivers keep the original contract, because their work can outlive the gateway.Also changed:
driver_operation_owner.Scope
This is the narrow fix from the design note. The broader proposal, making generation replacement fully level-triggered in the driver, is not part of this PR.
Tests
New, in
compute::tests:restart_abandoned_by_a_stopped_gateway_is_taken_over_and_retried: the incident. Nothing retries the sandbox before takeover. Once the owner's presence has not changed for the full delay, the operation is taken over and the sandbox is restarted (one stop, one start).heartbeating_owner_keeps_its_operation_until_its_presence_stops_changing: heartbeats reset the clock. Deletion is refused until the takeover, then succeeds.operation_without_a_recorded_owner_waits_the_longer_delaylate_response_from_the_previous_owner_cannot_settle_a_taken_over_operationheartbeat_that_hangs_fences_and_recovery_unfences: uses paused time. A hanging write fences within 60–75s, a recovered write unfences, and shutdown fences.fenced_gateway_abandons_the_driver_operation_it_owns: a process never takes over its own operation. Once fenced, it drops the blocked start, settles, and reschedules the restart.Results:
cargo test -p openshell-server --lib: 2179 passed.cargo clippy -p openshell-server -p openshell-cli --all-targets -D warnings: clean.The second commit addresses an adversarial review: hung heartbeats never fenced, wall-clock comparison, the rolling-upgrade owner wipe, unmanaged remote drivers, and fencing on shutdown.
🤖 Generated with Claude Code
https://claude.ai/code/session_014dzrkBzoiGwdAtxRmRxtR1