Only one smartos23-x64 executor online

[sxa]

We seem to have lost test-mnx-smartos23-x64-6 which means we're down to just one smartos23-x64 agent which is causing a bit of a backlog on some of the PR test runs.

[sxa]

@danmcd @jperkin I'm struggling to connect into the smartos machines just now (maybe a configuration error on my side) - can one of you take a look and see if we can get -6 back up again?

[danmcd]

There is a network tickle at this data center. Our ops folks are working on it.

[danmcd]

Still no luck with renumbering it yet. We may tear it down and rebuild it, depending. I have a much larger communication to send about SmartOS and illumos in general.

[Renegade334]

Any update? This is still proving to be quite a bottleneck.

[MikeMcC399]

As mentioned in https://openjs-foundation.slack.com/archives/C03BJP63CH0/p1787479247443919 this bottleneck is currently causing very long execution times of https://ci.nodejs.org/job/node-test-pull-request - up to 10 or 12 hours worst case.

The queue https://ci.nodejs.org/label/smartos23-x64/load-statistics has been saturated since Aug 16, 2026. It currently has a backlog of around 18 jobs with an approximate average execution time of 45 minutes / job.

---

relates to nodejs/jenkins-alerts#5992

[sxa]

There was a particular problem with the PR testers at the end of last week which has caused a backlog of jobs. This has been vastly more visible on smartos due to only having one executor and jobs taking 45 minutes each. Even after what would normally be a relatively quiet weekend period there are still about 15 jobs queued up at the time of writing (Morning morning UTC). On that basis I am going to disable the smartos23 jobs for today while the backlog is cleared as this is having a significant effect on the speed at which contributors can get PR results

[sxa]

Re-enabled smartos23 as queue is now empty.

[sxa]

Have disabled the https://ci.nodejs.org/job/node-test-commit-smartos/ job again (as opposed to deleting it from the node-test-commit configuration which I had before) as that's simpler to revert. It will show:

10:25:31     >> Job status: Skipping [node-test-commit-smartos]. This subjob has been disabled.

in the logs for node-test-commit and won't show it in the list of subjobs. Will re-enable later - we've got a bit of a backlog of jobs at the moment especially given that the fanned jobs were having issues with git-rebase today which - combined with the "resume build" option not always showing up means that some jobs are having to re-run on all platforms at the moment which is limited by the single smartos executor.

[panva]

Don't all the resumes that originally started with smartos still queue up to run it?

e.g. nodejs/node#65702 > https://ci.nodejs.org/job/node-test-pull-request/76859/

Despite it being disabled 20 hours ago a resume from 8 hours ago still waits for smartos to get picked up.

[sxa]

Don't all the resumes that originally started with smartos still queue up to run it?

e.g. nodejs/node#65702 > https://ci.nodejs.org/job/node-test-pull-request/76859/

Despite it being disabled 20 hours ago a resume from 8 hours ago still waits for smartos to get picked up.

Yes that seems to be correct but this provided a way of clearing the queue temporarily for new jobs which will give those resumes a chance of getting scheduled. Looks like Antoine and Filip cancelled a few in the last hour based on the load chart.