Skip to content

test(conformance): prompt delete when the workload is slow to handle TERM - #5291

Merged
elezar merged 2 commits into
mainfrom
test/5290-running-delete-deadline/matthewgrossman
Nov 6, 2026
Merged

elezar merged 2 commits into
mainfrom
test/5290-running-delete-deadline/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Summary

Many workloads do not exit immediately on TERM, for example while they run cleanup handlers. Add an all-driver conformance test requiring that deleting such a running sandbox still completes promptly, rather than waiting out the runtime's termination grace period.

Related Issue

Closes #5290.

Related to #5285, which remains open for stronger signal-processing and runtime-termination evidence. #5290 has no acceptance label; the author has verified ADMIN access.

Changes

  • Run a workload whose TERM handler waits TERM_HANDLING_DELAY (3 s) before exiting, and synchronize on a readiness marker written after the trap is installed.
  • Require deletion and absence from get/list within one shared DELETE_DEADLINE (10 s) that starts before deletion.
  • Document the lifecycle suite coverage.

Why these values:

  • 3 s delay: the workload must still be handling TERM when the boundary receives its own TERM. Locally that came up to ~0.85 s after the workload's TERM. The delay stays below the supervisor's 5 s wait before it kills the workload, so the test does not exercise that escalation path.
  • 10 s deadline: well below the 30 s Kubernetes grace period that a shutdown hang waits out, with headroom over the ~4 s normal deletion.

The assertion is prompt deletion, not graceful workload exit or proof that every runtime process avoids SIGKILL. OpenShell may kill the workload before its handler finishes.

Release Notes

Testing

  • Relevant checks pass: conformance Rust formatting, lifecycle Clippy with warnings denied, and git diff checks.
  • E2E test added. On local Linux arm64 k3s/containerd, using public deletion with only the boundary image varied:
TERM delay Baseline d7da0c7 Baseline plus #5279 boundary change
1 s 32.0 s, 32.4 s 4.1 s, 4.7 s
3 s 32.6 s, 31.4 s 4.1 s, 4.2 s

Other compute drivers and amd64 CI have not been exercised locally. The unchanged supervisor still waits for forced killing in this Kubernetes setup, even when public deletion completes within the deadline. This PR contains only the test, not the #5279 fix; it is expected to fail on builds without that fix.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

@matthewgrossman matthewgrossman changed the title test(conformance): bound running-delete with delayed TERM test(conformance): prompt delete when the workload is slow to handle TERM Nov 5, 2026
elezar
elezar previously approved these changes Nov 5, 2026

@elezar elezar left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@elezar
elezar force-pushed the test/5290-running-delete-deadline/matthewgrossman branch from fbcbfe9 to 3974537 Compare November 6, 2026 12:38
@elezar
elezar enabled auto-merge November 6, 2026 12:51
@elezar
elezar added this pull request to the merge queue Nov 6, 2026
Merged via the queue into main with commit 4a25a6b Nov 6, 2026
85 of 86 checks passed
@elezar
elezar deleted the test/5290-running-delete-deadline/matthewgrossman branch November 6, 2026 13:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:sandbox Sandbox runtime and isolation work topic:testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(conformance): bound running-delete with a one-second TERM delay

2 participants