Skip to content

illumos fixes for job process spawning - #103

Open
plotnick wants to merge 6 commits into
verify-chainfrom
illumos-fixes
Open

plotnick wants to merge 6 commits into
verify-chainfrom
illumos-fixes

Conversation

@plotnick

@plotnick plotnick commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #102 by launching each job in its own contract (like sshd does). This means that sush jobs can outlive sled-agent, and so, e.g., svcadm disable sled-agent will not kill running jobs. Verified fix on Dublin, where a reboot job was able to successfully reboot a sled. When it came back up (version all green), sush correctly reported the job as interrupted:

sush# job start -w reboot -T 2F8JEXDK
✅ Signed request for job `turkey-wreck-merry-talent-plate-food-salute-mean`
❌ Communication error: error sending request for url (https://link-local.sush:12352/jobs/turkey-wreck-merry-talent-plate-food-salute-mean/status?via=913-0000023%3A2F8JEXDK)
...
sush# job status turkey-wreck-merry-talent-plate-food-salute-mean
❌ Job ID:	turkey-wreck-merry-talent-plate-food-salute-mean
   913-0000023:2F8JEXDK  Error at 1986-12-28 00:01:08.035729813 UTC: The server restarted while the job was running

Also switches job launch from fork/exec to posix_spawn(3C), so that we don't copy sled-agent's entire address space and block its threads with every job launch.

Commit f13d246 is a work-around for illumos 18395, which was fixed in oxidecomputer/illumos-gate@460f7917fa4f but has not yet made it into buildomat images; see oxidecomputer/meta#1148. It should be reverted once that lands.

Commit bdc30df is a work-around for illumos 18596. It is perfectly safe on all platforms and may remain even after that issue is fixed.

@plotnick
plotnick force-pushed the illumos-fixes branch 2 times, most recently from 74c50f7 to 6a4c40e Compare October 24, 2026 16:58
@plotnick plotnick changed the title Spawn each job in a process contract of its own illumos fixes for job process spawning Oct 24, 2026
@plotnick
plotnick force-pushed the illumos-fixes branch 4 times, most recently from 24e04d4 to 8d5bf62 Compare October 25, 2026 05:28
plotnick and others added 4 commits October 25, 2026 09:35
Fixes #102.

Co-Authored-By: Claude <noreply@anthropic.com>
Before illumos 18395[1], files in ctfs fail open(2) with `EINVAL` when
`O_CLOEXEC` is set, and Rust's `std::fs` always sets it. So instead
we open the contract files with `rustix` and set `FD_CLOEXEC` after.

To be reverted once we're up to date with [2].

[1] https://www.illumos.org/issues/18395
[2] oxidecomputer/illumos-gate@460f7917fa4f

Co-Authored-By: Claude <noreply@anthropic.com>
Forking sled-agent to start a job copies its large address space and
stops all of its threads. On illumos, posix_spawn builds the child in
the kernel instead, and does neither.

posix_spawn runs none of our code in the child, so a small bash shim
sets the job's limits before it execs the job. JobLimits::clamp now
also lowers each limit to our own hard limit, which a job cannot
exceed. JobChild replaces tokio's Child, and reaps the job when
SIGCHLD arrives. Errors from libcontract and posix_spawn now go
through one check_errno, which names the call that failed.

Co-Authored-By: Claude <noreply@anthropic.com>
A stop used to signal the job's process group, so a descendant that
started its own process group escaped it. Signalling the job's
contract reaches every process in the job.

Co-Authored-By: Claude <noreply@anthropic.com>
On illumos, after a pty's subsidiary closes, polling the master for
both reading and writing can report nothing but a write error, even
though an EOF is ready to read. The job's pty reader was previously
registered for both directions, so it never saw the EOF, and the drain
timeout expired instead.

Co-authored-by: Claude <noreply@anthropic.com>
@plotnick
plotnick marked this pull request as ready for review October 26, 2026 04:14

@danmcd danmcd left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get the feeling a Rust front-end to ctfs(4FS) should be something available in a crate. Happy to be corrected, however.

Comment thread server/src/contract.rs Resolved
Co-authored-by: Claude <noreply@anthropic.com>

@danmcd danmcd left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pass 0 results: I'll be interested in seeing contract.rs get further shrunk or even moved into something shareable by others. For now, this is an improvement, thank you.

I need to give this pass 1 before I can approve. Pardon latency.

Comment thread server/src/contract.rs Resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants