SIGTERM (e.g. a SLURM time limit) leaves info.json empty #459

duct handles SIGINT but not SIGTERM. A duct process that gets SIGTERM dies immediately, so info.json is never written: it stays the empty file duct creates at startup. usage.jsonl keeps the samples written so far, but the run has no summary, and con-duct ls shows nothing for it. These are exactly the runs that most need a post-mortem: SLURM time limits, scancel, node drains, a plain kill.

Reproducer

con-duct run -p demo/ sleep 300 &
sleep 3; kill -TERM $!; wait $!; echo "duct exit: $?"
ls -l demo/

duct exit: 143
-rw-r--r--. 1 austin austin   0 Oct  5 15:23 info.json
-rw-r--r--. 1 austin austin   0 Oct  5 15:23 stderr
-rw-r--r--. 1 austin austin   0 Oct  5 15:23 stdout
-rw-r--r--. 1 austin austin 424 Oct  5 15:23 usage.jsonl

The sleep 300 child is also left running: duct does not pass the signal on, and in the default new-session mode the child is in its own session, so even a TERM to duct's process group misses it. (Reproduced on current main.)

Seen in the wild

An fMRIPrep job under SLURM (via babs), duct singularity run ..., --time 6-00:00:00, hit its time limit. On that cluster (ProctrackType = proctrack/cgroup, KillWait = 30 sec) slurmstepd logged CANCELLED ... DUE TO TIME LIMIT; with that config SLURM signals every process in the job's cgroup directly, TERM and then KILL 30 s later. Result: info.json 0 bytes with the job-start mtime, usage.jsonl (69 MB) and stdout/stderr intact, so six days of compute with no summary.

One detail we cannot yet explain: duct's last usage.jsonl record is 7 s after slurmstepd's cancel message (same node clock), so duct was still alive briefly after the cancel. Possibly just the gap between the message and the signal delivery.

Possible approach

Not a decided design, for discussion:

1. Handle SIGTERM. Forward it to the child (to its process group in new-session mode, so the whole tree gets it), then let the normal wait-and-finalize path run. This is also what Implement "attach" command and graceful exiting #407 (attach) assumes: a plain kill should shut duct down gracefully.
2. Do not depend on the child exiting in time. Under SLURM the child gets TERM too, and duct has only KillWait (often 30 s) before KILL; a large job will often not exit by then. So the handler could write a provisional info.json immediately on TERM, before forwarding and waiting, and overwrite it if the child exits in time.
3. Never leave info.json empty. Write it at startup with what is already known (command, start time, system info, a "running" marker) and rewrite it at the end, both atomically (temp file + rename). That also covers SIGKILL, OOM kills and node failures, which no handler can catch. This changes what readers of info.json (ls, plot) must expect, so it is the part that needs a decision.

Related: #184 (Ctrl-C handling, where the current SIGINT handler came from), #407 (attach and graceful exit), #130 (SLURM sacct info, the only summary left for such a run today).

Testing idea (from @yarikoptic)

A pytest fixture that brings up SLURM in a container (as babs's CI does with a slurm-docker image), submits a duct-wrapped job, and asserts info.json is written. SLURM's minimum --time is one minute, so either wait out a 1-minute limit (closest to the real case) or scancel --signal=TERM <job> a few seconds in, which tests the handler without waiting.

🤖 Generated with Claude Code