Skip to content

Fix #1459: Call check_signals in the scan to support signal handlers - #10101

Open
pamod-madubashana wants to merge 4 commits into
vortex-data:developfrom
pamod-madubashana:pamod-madubashana/issue-1459-36407257745
Open

pamod-madubashana wants to merge 4 commits into
vortex-data:developfrom
pamod-madubashana:pamod-madubashana/issue-1459-36407257745

Conversation

@pamod-madubashana

Copy link
Copy Markdown

Fixes #1459

What changed

Honours Python signal handlers (e.g. Ctrl+C / KeyboardInterrupt) while scanning from Python by calling py.check_signals() at iteration boundaries in vortex-python/src/iter/:

  • PyArrayIterator::__next__ now checks signals with the GIL held before and after the detached chunk fetch, so each next() raises promptly if a signal arrived.
  • read_all now pulls chunks one at a time, checking signals before each chunk (plus a final check), instead of running the whole scan inside a single detach with no signal checks.
  • The to_arrow RecordBatchReader closure checks signals via Python::attach before converting each chunk, so pyarrow readers are interruptible too.
  • PythonArrayIterator::next now implements the signal check its comment always promised, mapping a PyErr into a VortexError since only VortexResult is available there.

Why

Long scans run inside py.detach(...), which releases the GIL, so Python signal handlers never run and Ctrl+C appears to hang until the scan finishes. Checking signals wherever we already hold the GIL at chunk boundaries lets the interpreter raise KeyboardInterrupt between chunks without changing any scan semantics.

Verification

  • cargo fmt -p vortex-python -- --check: clean (only stable-channel warnings about nightly-only options, no diff).
  • cargo clippy -p vortex-python --all-targets -- -D warnings: could not compile in this environment — vortex-array's build script fails with failed to run flatc: No such file or directory (os error 2), a pre-existing missing FlatBuffers compiler unrelated to this change. For the same reason no Python test requiring the built extension could be run.

@robert3005

Copy link
Copy Markdown
Contributor

I think this needs some testing?

@pamod-madubashana

pamod-madubashana commented Sep 30, 2026 •

Copy link
Copy Markdown
Author

Added vortex-python/test/test_iter_signals.py covering all four check_signals sites: next(), read_all() (pending-signal early-abort plus a mid-scan SIGINT timing test), and the to_arrow record-batch reader. Each test queues a real SIGINT against a counting Python iterator and asserts the scan surfaces it promptly without consuming every chunk first. Note I could not run the suite locally (no built extension / flatc in this environment), so CI verification would be appreciated.

@pamod-madubashana
pamod-madubashana force-pushed the pamod-madubashana/issue-1459-36407257745 branch from 417c9e7 to 31f2719 Compare September 30, 2026 04:56
@github-actions

Copy link
Copy Markdown
Contributor

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days

@github-actions github-actions Bot added the stale This PR is stale and will be auto-closed soon label Oct 21, 2026
…l handlers

Signed-off-by: Pamod Madubashana <contact@pamod.is-a.dev>
Queue a real SIGINT and assert next(), read_all(), and the to_arrow
record-batch reader surface it promptly while aborting early, plus a
mid-scan interrupt test proving read_all does not run to completion.

Signed-off-by: Pamod Madubashana <contact@pamod.is-a.dev>
@pamod-madubashana
pamod-madubashana force-pushed the pamod-madubashana/issue-1459-36407257745 branch from 6e268ac to 08fe8f7 Compare October 23, 2026 16:26
@pamod-madubashana

Copy link
Copy Markdown
Author

Rebased onto latest develop and resolved the merge conflict in vortex-python/src/iter/mod.rs. Also fixed a type-inference error in PyArrayIterator::__next__ (annotated the py.detach closure as VortexResult<Option<PyArrayRef>>) and made test_iter_signals.py runnable (the original signal.raise_signal fired synchronously and aborted pytest; switched to background-thread delivery). Built the extension, ran cargo fmt/clippy (clean), and all 4 signal tests pass. Force-pushed the resolved branch.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale This PR is stale and will be auto-closed soon

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Call check_signals in the scan to support signal handlers

2 participants