Skip to content

Hard-exit the neuron when its forward loop stalls - #723

Closed
LandynDev wants to merge 1 commit into
testfrom
vali/hard-exit-on-stall
Closed

Hard-exit the neuron when its forward loop stalls#723
LandynDev wants to merge 1 commit into
testfrom
vali/hard-exit-on-stall

Conversation

@LandynDev

Copy link
Copy Markdown
Collaborator

The stall watchdog in neurons/validator.py and neurons/miner.py calls sys.exit(1), but a stalled forward thread is wedged inside a socket read that never returns. The interpreter then waits for that thread at shutdown (concurrent.futures atexit join), so the process never exits, docker's restart policy never fires, and the neuron sits dead with the container reported Up.

Seen three times on the testnet validator today: the relay's vault event poll blocks forever in websockets recv under async_substrate_interface (stack in the linked issue), the watchdog logs "exiting for restart", the thread stops, the process stays alive.

exit_for_restart() logs, arms a daemon Timer that calls os._exit(1) after 30 s, then raises SystemExit so the normal cleanup still runs when it can. Both neurons use it for the dead-thread and stalled-forward cases.

Stopgap only. The root cause (no read timeout on the vault subtensor websocket in allways/vault/client.py poll_events) is filed separately.

@LandynDev
LandynDev force-pushed the vali/hard-exit-on-stall branch from fcaefce to 38746c2 Compare September 5, 2026 02:19
@LandynDev

Copy link
Copy Markdown
Collaborator Author

Closing: the fix belongs on the RPC connection (timeout + reconnect on the vault poll, #724), not a hard-exit workaround.

@LandynDev LandynDev closed this Sep 5, 2026
@LandynDev
LandynDev deleted the vali/hard-exit-on-stall branch September 5, 2026 02:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant