fix(python): keep private-loop worker off Python during interpreter exit#2008
Merged
Conversation
The python.yml Examples job crashed flaky (SIGABRT, ~25% of runs) in langgraph_async_tool.py at process exit: the bashkit-py-loop worker thread wakes when the engine is gc'd — commonly inside Py_Finalize — and called Python::attach to close its asyncio loop. Attaching a fresh thread state during finalization fatals CPython with 'PyGILState_Release: thread state must be current when releasing'. Python::try_attach does not help: its finalization check is compiled only for Python >= 3.13 and Py_IsInitialized() still returns 1 during Py_FinalizeEx's GC on older versions (verified via core dumps). The worker exit path no longer touches Python at all: the loop's Py ref is dropped unattached (pyo3 defers the decref) and the loop is closed by asyncio's BaseEventLoop.__del__ when the decref runs, or reclaimed by the OS at process exit. Documented as TM-PY-030 variant (3). Verified: example aborted 6/30 runs before, 0/80 across two stress runs after; full bashkit-python pytest suite passes (700 passed, 1 skipped).
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
bashkit | 499a94c | Commit Preview URL Branch Preview URL |
Jun 10 2026, 01:32 AM |
There was a problem hiding this comment.
Pull request overview
Fixes a flaky interpreter-exit crash in the bashkit-py-loop private-loop worker by ensuring the worker thread’s shutdown path does not attach to Python during Py_Finalize GC/finalization, avoiding CPython fatal aborts at process exit.
Changes:
- Remove Python interaction on the private-loop worker thread exit path (drop the loop
Pyref unattached instead of calling intoasyncioto close it). - Document the additional TM-PY-030 variant (interpreter-exit attach crash) in the threat model.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| specs/threat-model.md | Documents TM-PY-030 variant (3) describing the interpreter-exit crash scenario and mitigation. |
| crates/bashkit-python/src/lib.rs | Stops attaching to Python on private-loop worker shutdown; drops the event loop reference without touching Python. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
1767
to
+1771
| and dropped the last `Arc<Runtime>`; tokio's default `Runtime::drop` joins in-flight | ||
| blocking tasks, and an abandoned (timed-out) callback task must re-attach to finish — | ||
| freezing the entire interpreter. The `PyRuntime` handle now shuts the runtime down | ||
| with `shutdown_background()` on last drop. Regression tests: | ||
| with `shutdown_background()` on last drop. (3) The private-loop worker thread called | ||
| `Python::attach` on its exit path to close its asyncio loop; the worker usually wakes |
Review feedback: the table row only described the two deadlock variants while the paragraph below documents the interpreter-exit SIGABRT too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes the remaining red on main after #2007: the
python.ymlExamples job crashed with SIGABRT (core dumped) inlanggraph_async_tool.pyat process exit — flaky, ~25% of runs (it passed on the #2007 PR run, failed on the main push run).Why
Core-dump analysis: the
bashkit-py-loopprivate-loop worker thread wakes fromrecv()when the callback engine is gc'd — which commonly happens insidePy_Finalize's GC pass — and calledPython::attachto close its asyncio event loop. Attaching a fresh thread state during interpreter finalization fatals CPython:Python::try_attachwas tried first and does not help: its finalization check is compiled only for Python ≥ 3.13, andPy_IsInitialized()still returns 1 duringPy_FinalizeEx's GC on older versions (confirmed with a second core dump showing the abort insidetry_attachitself).This race predates #2007 — it shipped with the private-loop worker redesign (#1918) — but was masked because every
python.ymlrun on main since June 6 was cancelled by subsequent pushes.How
The worker's exit path no longer touches Python at all: the loop's
Pyref is dropped unattached (pyo3 safely defers the decref) and the loop is closed by asyncio'sBaseEventLoop.__del__when the deferred decref runs, or reclaimed by the OS at process exit. Documented as TM-PY-030 variant (3) inspecs/threat-model.md.Tests
langgraph_async_tool.pyaborted 6/30 runs (and 2/5, 1/10 in other rounds). After: 0/80 across two 40-run stress rounds.cargo fmt --check/cargo clippy -p bashkit-python --all-targetsclean.Generated by Claude Code