Skip to content

test-install: Fix chunkah hang by removing host systemd/dbus leak, without breaking journal logging - #2392

Open
Johan-Liebert1 wants to merge 2 commits into
bootc-dev:mainfrom
Johan-Liebert1:test-install-fix
Open

test-install: Fix chunkah hang by removing host systemd/dbus leak, without breaking journal logging#2392
Johan-Liebert1 wants to merge 2 commits into
bootc-dev:mainfrom
Johan-Liebert1:test-install-fix

Conversation

@Johan-Liebert1

@Johan-Liebert1 Johan-Liebert1 commented Aug 19, 2026 •
edited by cgwalters
Loading

Copy link
Copy Markdown
Collaborator

priv-integration.sh was intermittently hanging at the chunkah build step (a nested podman run invoked via systemd-run -dP --wait).

The root cause isn't fuse-overlayfs performance -- it's that the outer sudo podman run --privileged --pid=host ... invocation bind-mounts the host's real /run/systemd and /run/dbus into the container. Every systemd-run call inside therefore talks to the host's actual systemd manager, which forks the wrapped process into the host's mount/IPC namespaces instead of the container's. When that escaped process is itself a podman run (the chunkah build), it contends forever on host-level podman locks (/dev/shm/libpod_lock) against the real host-level sudo podman run --privileged process that's still running the CI job -- a deadlock.

This was confirmed empirically by comparing /proc/<pid>/ns/mnt/ns/ipc of the hung process against host PID 1, and reproduced reliably in isolation.

Update: an earlier version of this PR simply dropped both /run/systemd and /run/dbus and deleted the systemd-run wrappers. That fixes the hang, but a full end-to-end run turned up a second, unrelated issue it introduced: a few lines above the chunkah step, the script asserts on a "layers already present" message via nsenter -m -t 1 journalctl _COMM=bootc. ostree-ext's structured logging silently skips writing to the journal at all unless libsystemd::daemon::booted() (a check for /run/systemd/system) is true, so without /run/systemd that check has nothing to find and fails -- not a hang, just a different, real regression.

The fix now keeps /run/systemd mounted (only /run/dbus is dropped -- nothing here ever needed it) so that check keeps working, and has priv-integration.sh itself umount /run/systemd right after that check, before any of the systemd-run-free podman/build work later in the script. Cgroup delegation for nested podman run turns out to be re-evaluated dynamically on each invocation based on whatever systemd sockets are currently visible, not fixed once at outer-container-creation time, so this mid-script unmount is enough: with no live systemd socket at that point, podman's own cgroup-manager detection falls back to cgroupfs automatically and the nested chunkah container stays properly inside the outer container's own namespaces. None of the four systemd-run -dP --wait call sites need systemd supervision -- they're synchronous builds/copies that were only ever wrapped to get a cgroup for the nested container, which podman now arranges on its own.

Also keeps the --log-level=debug/RUST_LOG=chunkah=debug logging from the original version of this PR as a separate commit, since it's independently useful regardless of this fix.

Assisted-by: AI
I reviewed this investigation and the resulting patch; the root cause (and this later correction to it) was confirmed via repeated CI reproduction in a sandbox repo, including a full unmodified end-to-end run of priv-integration.sh with the exact patch applied, before landing here.

@github-actions github-actions Bot added the area/ostree Issues related to ostree label Aug 19, 2026
@bootc-bot
bootc-bot Bot requested a review from ckyrouac August 19, 2026 06:40
cgwalters added a commit to Johan-Liebert1/bootc that referenced this pull request Aug 20, 2026
@cgwalters
test-install: Enable debug logging for chunkah build step
175d957 
Turn on `podman --log-level=debug` and `RUST_LOG=chunkah=debug` for
the chunkah build invocation in priv-integration.sh. This step
converts the FCOS test image into a non-ostree image and previously
had no visibility into podman or chunkah internals when something
went wrong here (as with the host-namespace-escape hang fixed in the
prior commit). Cheap to leave on permanently since chunkah's own
progress output is otherwise sparse, and useful for any future
regressions in this step regardless of cause.

This was originally proposed as part of PR bootc-dev#2392 alongside a
(different, since superseded) fix attempt for the same underlying
hang; keeping it here since it's independently useful.

Assisted-by: AI
Signed-off-by: Colin Walters <walters@verbum.org>
@cgwalters cgwalters changed the title test-install: Use native overlay test-install: Fix chunkah hang by removing host systemd/dbus leak Aug 20, 2026
cgwalters added a commit to Johan-Liebert1/bootc that referenced this pull request Aug 20, 2026
@cgwalters
test-install: Enable debug logging for chunkah build step
39610d2 
Turn on `podman --log-level=debug` and `RUST_LOG=chunkah=debug` for
the chunkah build invocation in priv-integration.sh. This step
converts the FCOS test image into a non-ostree image and previously
had no visibility into podman or chunkah internals when something
went wrong here (as with the host-namespace-escape hang fixed in the
prior commit). Cheap to leave on permanently since chunkah's own
progress output is otherwise sparse, and useful for any future
regressions in this step regardless of cause.

This was originally proposed as part of PR bootc-dev#2392 alongside a
(different, since superseded) fix attempt for the same underlying
hang; keeping it here since it's independently useful.

Assisted-by: AI
Signed-off-by: Colin Walters <walters@verbum.org>
@cgwalters cgwalters changed the title test-install: Fix chunkah hang by removing host systemd/dbus leak test-install: Fix chunkah hang by removing host systemd/dbus leak, without breaking journal logging Aug 20, 2026
cgwalters added a commit to Johan-Liebert1/bootc that referenced this pull request Aug 20, 2026
@cgwalters
test-install: Enable debug logging for chunkah build step
f7f2d4e 
Turn on `podman --log-level=debug` and `RUST_LOG=chunkah=debug` for
the chunkah build invocation in priv-integration.sh. This step
converts the FCOS test image into a non-ostree image and previously
had no visibility into podman or chunkah internals when something
went wrong here (as with the host-namespace-escape hang fixed in the
prior commit). Cheap to leave on permanently since chunkah's own
progress output is otherwise sparse, and useful for any future
regressions in this step regardless of cause.

This was originally proposed as part of PR bootc-dev#2392 alongside a
(different, since superseded) fix attempt for the same underlying
hang; keeping it here since it's independently useful.

Assisted-by: AI
Signed-off-by: Colin Walters <walters@verbum.org>
@cgwalters @Johan-Liebert1
ci: Stop leaking host systemd/dbus into install-tests, drop systemd-run
430b811 
The "Test install" job's `priv-integration.sh` step hangs
intermittently at the chunkah build (a nested `podman run` invoked via
`systemd-run -dP --wait`). The root cause is not fuse-overlayfs
performance or the chunkah build itself: it's that the outer
`sudo podman run --privileged --pid=host ...` invocation bind-mounts
the HOST's real systemd and dbus sockets in
(`-v /run/dbus:/run/dbus -v /run/systemd:/run/systemd`), so every
`systemd-run` call inside the container talks to the host's actual
systemd manager rather than anything scoped to the container.

`systemd-run --wait` asks systemd to fork and supervise the wrapped
command, and with the host's real systemd manager on the other end of
that dbus call, the forked process ends up living in the HOST's mount
and IPC namespaces instead of the container's -- despite having been
launched from inside the privileged container. When that escaped
process is itself a `podman run` (as with the chunkah build), it
starts contending on host-level podman locks (e.g.
`/dev/shm/libpod_lock`) against the *actual* host-level `sudo podman
run --privileged` process that is itself still running the CI job,
which deadlocks forever.

This was tracked down empirically: comparing `/proc/<pid>/ns/mnt` and
`/proc/<pid>/ns/ipc` of the hung process against host PID 1 confirmed
the escape, and reproduced the hang reliably in isolation by stripping
priv-integration.sh down to just the chunkah step. None of the four
`systemd-run -dP --wait` call sites here actually need systemd
supervision: they're synchronous builds/copies (`podman build`, two
`skopeo copy` calls, and the chunkah `podman run`) that work fine run
directly, and were apparently only wrapped in systemd-run to get a
cgroup for the nested container.

Fixing this isn't just a matter of deleting those wrappers, though.
`/run/dbus` can go entirely -- nothing here ever dials it once
systemd-run is gone -- but `/run/systemd` has to stay mounted, because
the `nsenter -m -t 1 journalctl _COMM=bootc` check a few lines above
this depends on it: ostree-ext's structured logging silently skips
writing to the journal at all unless `libsystemd::daemon::booted()` (a
check for `/run/systemd/system`) is true, and this test's own "layers
already present" assertion relies on that write happening. But leaving
/run/systemd mounted for the whole script also breaks cgroup
delegation for the nested podman invocations below (podman's
cgroup-manager auto-detection only produces a working, fully delegated
`cgroupfs` setup when it can't see a live systemd socket at all) --
and that turns out to hold regardless of `--cgroup-manager` flags at
any level, since it's decided dynamically per invocation rather than
once at container-creation time.

So instead: keep /run/systemd mounted at container creation for the
journal check, then have priv-integration.sh itself unmount it right
after that check, before any of the systemd-run-free podman/build work
that follows. Verified with the same namespace comparison that the
nested chunkah process's mnt/ipc namespaces match the outer
container's own entrypoint rather than the host's, and with a full,
unmodified end-to-end run of priv-integration.sh that both the journal
check and the final "ok privileged integration" line pass.

Assisted-by: AI
Signed-off-by: Colin Walters <walters@verbum.org>

Signed-off-by: Pragyan Poudyal <pragyanpoudyal41999@gmail.com>
@cgwalters @Johan-Liebert1
test-install: Enable debug logging for chunkah build step
3f2bbe9 
Turn on `podman --log-level=debug` and `RUST_LOG=chunkah=debug` for
the chunkah build invocation in priv-integration.sh. This step
converts the FCOS test image into a non-ostree image and previously
had no visibility into podman or chunkah internals when something
went wrong here (as with the host-namespace-escape hang fixed in the
prior commit). Cheap to leave on permanently since chunkah's own
progress output is otherwise sparse, and useful for any future
regressions in this step regardless of cause.

This was originally proposed as part of PR bootc-dev#2392 alongside a
(different, since superseded) fix attempt for the same underlying
hang; keeping it here since it's independently useful.

Assisted-by: AI
Signed-off-by: Colin Walters <walters@verbum.org>
Johan-Liebert1
Johan-Liebert1 commented Aug 21, 2026
View reviewed changes

@Johan-Liebert1 Johan-Liebert1 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, can't approve my own PR though

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ckyrouac ckyrouac Awaiting requested review from ckyrouac

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

area/ostree Issues related to ostree

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Johan-Liebert1 @cgwalters