`bootc upgrade` with composefs self-deadlocks before returning

[jmpolom]

Source version analyzed: bootc v1.16.7
Tested operating system: Fedora 44

Summary

On a system installed with bootc's native composefs backend, bootc upgrade
successfully pulls and stages an update but never returns. The process blocks on
an exclusive flock held back by its own shared flock on /sysroot/composefs.

Reproduction and evidence

1. Boot a native-composefs deployment.

2. Publish a changed image to its tracked image reference.

3. Run sudo bootc upgrade.

4. After the image layers are downloaded, the last visible output is similar to:

   Fetching config sha256:11d4d65a...
   Storing manifest sha256:b82aa3bddae10f579699a41178b305d9b896a6fe1ca161675657b26ce351bdb4
   

5. From another console, bootc status reports that same manifest as the staged
   deployment, including its verity hash and timestamp. This demonstrates that
   image generation, kernel/initrd preparation, BLS staging, and
   write_composefs_state() completed.

6. The /usr/bin/bootc upgrade process remains asleep with no change in
   /proc/<pid>/io. Its wait channel is:

   locks_lock_inode_wait
   

7. lslocks reports two locks belonging to the same bootc PID and path:

   FLOCK  READ    /sysroot/composefs
   FLOCK  WRITE*  /sysroot/composefs  blocker=<same bootc PID>
   

The process therefore holds a shared repository lock while waiting indefinitely
for an exclusive lock on the same inode.

Behavior after interrupting the hung upgrade

The hung bootc process was terminated after confirming the self-blocking flock.
The staged deployment remained present and readable by bootc status. A
subsequent attempt to apply it returned without rebooting:

$ sudo bootc upgrade --apply
No changes in staged image: <image-reference>

This shows that the original upgrade completed the image import, boot artifact
and BLS preparation, and staged-state write before deadlocking in post-stage
garbage collection. It also suggests that the UpdateAction::Skip path for an
already-staged image returns before honoring --apply.

Source analysis

In crates/lib/src/bootc_composefs/update.rs, do_upgrade() retains the
repo returned by pull_composefs_repo() and the candidate mounted_fs while
it prepares the BLS/UKI boot artifacts and writes the staged state. It then
calls composefs_gc() through the separately opened booted_cfs.repo without
explicitly releasing the first repository handle.

GC begins by calling composefs_oci::upgrade_repo() in
crates/lib/src/bootc_composefs/gc.rs, which requires the conflicting write
lock. The observed self-blocking flock is consistent with the pulled repository
handle still retaining its read lock.

Proposed fix

Release the candidate mount and pulled repository after boot setup, before
entering GC:

let boot_digest = match boot_type {
    // existing BLS/UKI setup
};

drop(mounted_fs);
drop(repo);

// write staged state, then run composefs_gc()

An inner scope around the pull/mount/boot-setup work would provide the same
lifetime boundary. A regression test should verify that a composefs upgrade can
stage a changed image and complete post-stage GC without waiting on its own
repository lock.

---

AI assistance: OpenAI Codex (GPT-5) helped analyze the observed runtime
evidence, review the bootc v1.16.7 source, and draft this report.

[cgwalters]

Thanks, yes this was previously reported in #2364

We had a lot of CI issues over the past week, which are finally resolved and we're about to ship #2383 which should fix this!

[jmpolom]

When can we expect that to land in F44?

[jeckersb]

When can we expect that to land in F44?

It's building now, assuming everything is successful it should show up in testing soon