CI: every Build run today fails on the same three OS-sandbox tests, blocking merges #823

What happens

Every Build run on 2026-10-20 has failed, both of them on the same three OS-sandbox tests, in the generate test generate coverage step of .github/workflows/go.yml:

--- FAIL: TestLoadArtifactSandbox (63.39s)
    artifact_sandbox_test.go:36:
        Error: directory "/tmp/executor-1855460048/codefly-uds-32972-1576387846" exists
        Test:  TestLoadArtifactSandbox
FAIL	github.com/codefly-dev/core/agents/manager	93.048s

panic: test timed out after 10m0s
	running tests:
		TestE2E_OSSandbox_BlocksWriteOutsideAllowedPaths (9m35s)
FAIL	github.com/codefly-dev/core/toolbox/launch	600.206s

panic: test timed out after 10m0s
	running tests:
		TestSessionProductionAdmissionUsesRealSandbox (9m58s)
FAIL	github.com/codefly-dev/core/toolbox/session	600.339s

Two cleanup warnings appear beside the first failure and look like the same underlying problem:

(WARN) (AgentConn.Close) artifact executor cleanup failed error=context deadline exceeded
(WARN) (AgentConn.Close) artifact executor cleanup failed error=unlinkat /tmp/TestArtifactShutdownReportsSnapshotCleanupFailure4031631369/002/codefly-executor-1635162763: permission denied

The two hung tests ran to the 10-minute package timeout rather than failing an assertion, so the processes were not making progress, not merely slow.

Why this is the runner and not a diff

Two independent PRs built today fail identically:

Neither diff touches toolbox/launch, toolbox/session, or the sandbox paths under test, and #819 cannot plausibly hang a bubblewrap sandbox. All three failures are OS-sandbox (bubblewrap) tests, which is the common thread.

The split by date is clean — every green Build predates today, and nothing from today has passed:

main is green at b81d92e6, but its last run was 2026-10-18, so main has not been exercised since the window opened. Something changed between 2026-10-19 and 2026-10-20 — a runner image roll, a bubblewrap or kernel change on the hosted runner, or a merge in that window.

Not this

• Not build(deps): bump goreleaser/goreleaser-action from 7.2.3 to 7.3.0 in the github-actions group #818. It is also red, but for a different and legitimate reason: it changes .github/workflows/go-service-release.yml, so internal/ciguard refuses it (TestTheDecisionRefusesAChangedWorkflow, TestEveryPermittedJobMatchesItsRecordedShape, and three more). That is the guard working as designed — see .claude/skills/ci-guard-triage. It needs its recorded job shape updated, which is a separate piece of work.
• Not a local failure. go test ./... is fully green on macOS (exit 0, zero FAIL lines), as is go test -race on the affected packages. The sandbox tests that hang on the runner are not reproducible there, which is consistent with a Linux/bubblewrap-specific cause.

Worth checking first

1. The permission denied on unlinkat under /tmp/Test…/codefly-executor-… suggests the sandbox is leaving files the test process cannot remove — a uid/mapping or mount-propagation change in the runner's bwrap would do that, and it would explain both the "directory exists" assertion and the hangs (a sandbox that cannot be torn down blocking the next one).
2. TestLoadArtifactSandbox asserts a /tmp/executor-*/codefly-uds-* directory does not exist. If a previous test's sandbox now survives cleanup, this fails for reasons unrelated to what it is testing — so the leak is likely upstream of the assertion.
3. Whether the 10-minute package timeout is simply too tight for these tests under coverage instrumentation, or whether they are genuinely deadlocked. The two numbers (9m35s and 9m58s) are suspiciously close to the limit, but a step that used to finish well inside it does not drift there without a cause.

Impact

No core PR can merge on green until this is resolved, which currently includes #821 and #819 and will include every new PR. Re-running the job is worth one attempt to rule out a one-off, but two independent PRs failing the same way on the same day argues against a flake.

Not verified

I did not bisect the window, did not re-run either job to test reproducibility, and did not inspect the runner image diff. I have not reproduced any of the three failures locally (macOS, where the suite is green). This issue reports a correlation across two runs with the full logs linked, not a root cause.