[schickling-assistant]

Problem

A dependency-evaluation/check scope launched about 220 concurrent buck2-fingerprint processes over dependency trees, causing ZFS dnode-lookup livelock and load around 320.

Two source paths multiply subprocesses:

• buck2/dependencies/defs.bzl exports transitive read roots; buck2/materialization.bzl unions them into PackageTreeInfo.read_roots. TypeScript, JavaScript and package-command actions call hashDeclaredInputRoots before/after execution. Its root-wide Promise.all starts one native process per root, multiplied by concurrent local Buck actions.
• Editor publication prepares up to four candidates, but each candidate's declared-root and payload checks use root-wide Promise.allSettled. Four candidates did not bound fingerprint subprocesses.

Rust walkers are synchronous recursive walks with one streaming buffer per walk: no Rayon, worker threads or CPU-scaled internal parallelism. These paths explain hundreds of subprocesses; exact incident PID ancestry was not captured by this change.

Goal

At most four cooperating native fingerprint walks per host, independent of CPU count, daemon, action, worktree, publisher or executable realization. Avoid unbounded fingerprint child creation within each TypeScript process.

Decisions

• Enforce the fixed cap inside the native library, covering both fingerprint modes and direct callers. Four slot locks coordinate processes; waiting blocks in kernel locks, never polling. Descriptor lifetime releases capacity, including after process death.
• Retain the semaphore rather than Buck-only resource limits: separate Buck daemons/worktrees and direct publishers cannot share a daemon-local resource.
• Coordinate through one fixed host directory independent of TMPDIR, user and executable realization. Native initialization prepares uniquely named temporary directories/files, sets modes 1777/0666 before publication, then publishes directories by rename and files by no-replace hard link. Losing contenders use the winner and remove temporary entries. This closes the cold-start create-before-chmod race.
• Existing permission bits need not match exactly. Actual read-only open and lock operations determine usability; failures name the path and required sharing mode. Retain regular-file/symlink safety checks. Read-only descriptors remain compatible with the sandbox's read-only mount.
• Admission serializes free-slot scans only. Full-cap waiters release admission and block on slot pid % 4, distributing waiters rather than retaining admission behind slot zero.
• Linux actions have private /tmp. The launcher initializes persistent host files without truncation and read-only binds their directory into each sandbox; otherwise a /tmp semaphore would be per action. No writable per-file mounts were added.
• The shared TypeScript runner queues one child per process; failures do not poison later requests. Existing all-settled candidate lifetime and deterministic error handling remain intact.
• Coordination deliberately stays outside action digests. It changes scheduling, not declared inputs, content hashing, stability checks or successful output bytes. Its unused file contents are not fingerprint inputs. README documents this boundary.

Verification

Prior lightweight local proof on initial commit 4c2084ce15bbb5a64f224e0f4d49035040f42ce4, using Bun 1.4.2:

bun test packages/@overeng/buck2-tools/src/typescript-runner.unit.test.ts --test-name-pattern 'queues root subprocesses'
1 pass; 15 filtered out; 0 fail; 4 expect() calls; 41 ms

This uses event/stream doubles, not fingerprint executions. An initial test-double attempt timed out; replacing direct ChildProcess construction produced the passing result. Subsequent review revisions, including head e02ff4bd04f2de4949d7d4622181635cb7a754d2, ran only scoped pinned oxfmt and Rust 1.98.1 rustfmt locally.

Rust coverage added, execution awaiting hosted CI:

• Twelve simultaneous cold-start callers, with no warm-up: every acquisition succeeds, and losing initializers leave no temporary entries.
• Seven contenders with maximum active count exactly four.
• Full admission path with deterministic PID and a synchronization hook after the full scan: release slot one while slot zero remains locked, and the waiter enters. No timing sleeps or scheduling guesses.
• Subprocess initialization under umask 077, exact new-mode assertions, shared slots across processes, and capacity release on child termination.
• Existing usable 0700/0600 modes are accepted; symlinked coordination files are rejected.

The sandbox probe checks lock-file visibility and read-only containment without imposing exact modes on existing paths. No Cargo, Buck build/test, fingerprint execution, full check or heavy proof ran on quarantined development hosts. Native, integration, full TypeScript and generated-freshness proof remains for hosted CI. genie:run was skipped because editor bootstrap requires prohibited Buck execution.

The existing Cargo CI lane runs cargo:test:buck2-foundation (cargo test --locked --package 'buck2-*'), Clippy and format checks. Its inherited heavyCiIf admits only merge-group/main-push events: ordinary draft PRs and branch workflow_dispatch skip Cargo. Rust unit-test execution therefore remains deferred to an admitted hosted lane; no green native proof is claimed.

Before → after (contract, not benchmark): unbounded subprocess fan-out and no host walk limit → one child per TypeScript process, at most four cooperating native walks across the host.

Complexity

One dependency-free admission module, one process-local promise queue and a read-only coordination mount in the existing action launcher. Temporary-name atomic publication is necessary to prevent concurrent cold starts from seeing restrictive intermediate permissions; shared inodes are necessary across separate mount namespaces.

Concerns

• A PID-selected waiter can wait while a different slot is idle; bounded filesystem pressure takes priority over throughput.
• Older binaries must be quiesced/upgraded. Other mount namespaces must expose the same host directory.
• Never unlink active coordination files. Actual permission/access failures stop admission rather than bypassing the cap.
• Hosted proof remains pending; this PR remains draft.

Friction & bottlenecks

• Heavy local proofs are prohibited; the existing hosted Cargo lane also excludes draft PRs and branch dispatch.
• Used pinned oxfmt's documented passthrough for scoped formatting without invoking quarantined task dependencies.
• The initial Bun test double timed out; replaced direct ChildProcess construction with an event/stream double.

Follow-ups

None added to scope. Existing overlapping work: #1795 (fingerprint capability isolation), #1831 (CAS publication reuse). This change uses current main without adopting their pending changes.

References

Filesystem-pressure incident reported 2026-10-15; no public incident issue. Related: #1795, #1831.

[github-actions]

Pipeline traces

20 jobs; 5 successful, 0 failed, 0 cancelled, 15 skipped

Job	Status	Wall time	Delta vs main p50	Trace
source-shape	success	0m 10s	no main baseline	not instrumented
tested-tree	success	0m 04s	no main baseline	not instrumented
pr-reviews-resolved	success	0m 04s	no main baseline	not instrumented
default-ref-policy	success	0m 58s	no main baseline	not instrumented
quality	success	9m 54s	no main baseline	Explore

All other jobs (15)

Job	Status	Wall time	Delta vs main p50	Trace
trusted-buck2-remote-cache-proof	skipped	unavailable	duration unavailable	unavailable
publish-products	skipped	unavailable	duration unavailable	unavailable
deploy-storybooks	skipped	unavailable	duration unavailable	unavailable
test-integration-notion	skipped	unavailable	duration unavailable	unavailable
test-live-deploy-ci-tools	skipped	unavailable	duration unavailable	unavailable
test-playwright-utils	skipped	unavailable	duration unavailable	unavailable
pr-a-inert-buck	skipped	unavailable	duration unavailable	unavailable
build-products	skipped	unavailable	duration unavailable	unavailable
cargo	skipped	unavailable	duration unavailable	unavailable
test-integration-restate	skipped	unavailable	duration unavailable	unavailable
test-playwright-tui-react	skipped	unavailable	duration unavailable	unavailable
test[runner=namespace-profile-macos-arm64]	skipped	unavailable	duration unavailable	unavailable
test[runner=namespace-profile-linux-x86-64]	skipped	unavailable	duration unavailable	unavailable
weaver	skipped	unavailable	duration unavailable	unavailable
notify-alignment	skipped	unavailable	duration unavailable	unavailable

Main baseline samples (by job): source-shape n=0, tested-tree n=0, pr-reviews-resolved n=0, default-ref-policy n=0, trusted-buck2-remote-cache-proof n=0, publish-products n=0, deploy-storybooks n=0, test-integration-notion n=0, test-live-deploy-ci-tools n=0, quality n=0, test-playwright-utils n=0, pr-a-inert-buck n=0, build-products n=0, cargo n=0, test-integration-restate n=0, test-playwright-tui-react n=0, test[runner=namespace-profile-macos-arm64] n=0, test[runner=namespace-profile-linux-x86-64] n=0, weaver n=0, notify-alignment n=0.
Selected main run IDs: none.
Task-level durations are not included. Trace links may be empty while export, indexing, or retention is pending.

15 job(s) without a start time are omitted from the timeline, not measured as zero.