Add erofs support to apko. - #2249
Open
smoser wants to merge 27 commits into
Open
Conversation
smoser
force-pushed
the
feat/apko-erofs
branch
4 times, most recently
from
May 27, 2026 15:29
4516981 to
268396a
Compare
Contributor
Author
raharper
reviewed
May 27, 2026
smoser
force-pushed
the
feat/apko-erofs
branch
2 times, most recently
from
May 27, 2026 19:54
2101db3 to
3af7df1
Compare
This was referenced Aug 1, 2026
Contributor
Author
|
I think this is ready for some eyes to review and some testing of it. See the description and linked PRs that i put together to show its usefulness. |
raharper
reviewed
Aug 3, 2026
raharper
left a comment
raharper
left a comment
There was a problem hiding this comment.
Looks really solid and complete with extensive testing. a couple comments/questions I'm interested in see feedback.
Emit OCI image layers as EROFS filesystem images (application/vnd.erofs) instead of tar+gzip. Selected via `--format=erofs` on `apko build` / `apko publish` or `format: erofs` in apko.yaml. Tracks the draft erofs/erofs-image-spec (PR chainguard-dev#1). Single-layer and multi-layer (layering) builds are supported. Multi-layer emits each non-final group with `org.erofs.role=overlay-lower` per spec §3.8 and a per-group partial `usr/lib/apk/db/installed` so per-layer scanners still work. Manifests declare `erofs` in os.features per §5.4. Uses github.com/erofs/go-erofs (Apache-2.0, pure Go) for the writer. Reproducibility via SOURCE_DATE_EPOCH. Tests cover roundtrip via erofs.Open, byte-identical determinism, the full ImageLayoutToLayer dispatch, OSFeatures plumbing, and end-to-end validation via `fsck.erofs` (skipped when the binary isn't on PATH). `+zstd`, dm-verity, and chunk indexes are not implemented in this round. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Step-by-step guide for producing EROFS images with --format=erofs, inspecting the layer blob without root (fsck.erofs / dump.erofs / fsck.erofs --extract), mounting it (kernel mount or erofsfuse), pulling layer blobs from a registry, and assembling multi-layer images via overlayfs. Includes the current limitations (no +zstd, dm-verity, chunk index) and links from apko_file.md. All commands shown were verified against a real `apko build` of examples/wolfi-base.yaml. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds an `apko erofs` command group that wraps the EROFS mount workflow: `mount` accepts a raw blob or an OCI image directory (auto-detected, or via `erofs:`/`oci:`/`oci-dir:` prefixes), `umount` reads a per-mount state file to unwind every layer, and `ls` produces a `tar tvf`-style listing without leaving mounts behind. The new pkg/erofsmount library handles source parsing, OCI layout reading, kernel/FUSE drivers with kernel-overlay-over- fuse fallback to fuse-overlayfs, and state-file teardown. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`apko erofs ls` now opens each EROFS layer blob directly with go-erofs and walks a layered fs.FS in user space, instead of mounting the layers and walking the merged mountpoint. This removes the kernel/FUSE dependency for `ls` (works on darwin/windows too), eliminates the mount log noise, and is faster. Introduces a reusable pkg/erofsmount.Stack: a layered fs.FS implementing fs.ReadDirFS/StatFS/ReadLinkFS with full AUFS-style overlay semantics — .wh.NAME whiteouts hide siblings, .wh..wh..opq markers hide all lower- layer entries in a directory, ancestor whiteouts hide whole subtrees, type-mismatch in a higher layer shadows lower contents. apko's writer never emits whiteouts (it splits one rootfs into groups, doesn't merge), so 15 unit tests synthesize the whiteout cases via testing/fstest.MapFS. Mount and Unmount remain Linux-only since they genuinely need the kernel or FUSE. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
go-erofs cannot write compressed EROFS images yet, so the new compound format value routes compressed builds through `mkfs.erofs` (erofs-utils). ALGO is one of zstd|lz4|lz4hc|deflate, with an optional ,level=N. Plain `--format=erofs` keeps using the pure-Go writer. LayerFormat gains Base/Compressor/CompressionLevel methods; Valid is extended to whitelist the compressor names mkfs.erofs supports. Existing dispatch in build.go/layers.go/oci/image.go switches from Resolved() to Base() so the compressor suffix doesn't break format-kind comparisons. `apko erofs ls` wraps go-erofs's ErrNotImplemented (returned for compressed images on the read side) with a friendly message pointing the user at `apko erofs mount`, which decompresses via the kernel or erofsfuse. This is expected to be temporary — once go-erofs gains read-side compression, `apko erofs ls` will work against compressed images without code changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
writeERofs was the only identifier in the repo using mid-word
acronym-style "ERofs"; everywhere else treats it as a word ("Erofs").
Rename writeERofs / writeERofsViaMkfs and the related test names so the
codebase is uniform.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…d layers The mkfs.erofs path previously returned the same SHA-256 for Digest and DiffID and read its `Uncompressed()` bytes from the compressed file, which is wrong for OCI: `rootfs.diff_ids` is supposed to identify the uncompressed layer payload, and `org.erofs.uncompressed-digest` was declared as a constant but never set on any descriptor. Run mkfs.erofs a second time without `-z` to materialize the uncompressed-equivalent image, hash it for DiffID, persist it for the lifetime of the returned `v1.Layer`, and surface the digest via the spec's `org.erofs.uncompressed-digest` annotation. Raw EROFS layers keep DiffID == Digest as before. Also fix a double-close in `ImageLayoutToLayer`: the mkfs branch explicitly closed the output file but the deferred close at the top of the function ran again on the same descriptor. Move the defer past the mkfs branch so it only registers for paths that hold the descriptor. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
apko_file.md still claimed EROFS layers were uncompressed and that +zstd was unimplemented; the +ALGO variants have shipped since the format field was first documented. Rewrite the format list to enumerate raw and compressed variants, mention the uncompressed-digest annotation, and note the mkfs.erofs runtime dependency. erofs.md's manual-overlay reference snippet had two bugs that prevented it from running end-to-end: $ROOT/../../blobs/sha256/$MANIFEST double-traversed the OCI layout, and the lowerdir chain hard-coded a four-layer count with explicit lower0/lower1 references that wouldn't generalize. Rewrite the loop to derive $BLOBS and $MANIFEST cleanly and accumulate $LOWERS as it mounts. Also fix two small accuracy bugs: --arch on apko takes Go arches (amd64, arm64), not uname -m output (x86_64, aarch64) — replace with --arch=host, which is what the YAML examples in the same file use. And on Debian/Ubuntu erofsfuse ships inside the erofs-utils package; the separate erofsfuse package only exists on Wolfi/Alpine. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The --format help on apko build and apko publish previously listed only 'tar' and 'erofs', so users had no way to discover the erofs+zstd|lz4|lz4hc|deflate[,level=N] variants from --help. Spell out the compound form and the supported algorithms. LayerFormat.Valid() also silently accepted any "key=value" trailing options as long as the compressor name parsed: erofs+zstd,level=oops returned valid with level dropped, and erofs+zstd,foo=bar (or any unknown key) was treated as fine. That meant typos produced uncompressed fallback at best and surprising mkfs.erofs invocations at worst. Tighten Valid() to require known keys (currently just "level") with parseable values, and update the table-driven test to cover unknown keys, bare options without '=', and the previously-accepted level=BAD case. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The application/vnd.erofs media type and the org.erofs.role / overlay-lower / org.erofs.uncompressed-digest annotation strings lived as parallel unexported consts in pkg/build/erofs.go and pkg/erofsmount/oci.go with a "keep in sync" comment guarding the duplicate. Promote them to a single set of exported constants in pkg/build/types/erofs.go so both the writer and the reader/mount tools reference the same source, and test fixtures lock to the same strings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mount(8) since util-linux 2.29 autodetects when the source is a regular file and allocates a loop device with O_AUTOCLEAR, freeing it on umount. Asking for "-o loop" explicitly relies on a separate code path whose cleanup semantics differ across util-linux releases and busybox builds — on older or non-GNU versions the loop device can leak after umount. Drop "loop" from the argv. Keep "-o ro" to document intent (EROFS is intrinsically read-only, but the explicit flag tells a reader who is copy-pasting the equivalent shell command that we never plan to write). Update the matching tests and the two "doing it manually" snippets in docs/erofs.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
st.Mounts is recorded overlay-first then per-layer mounts in LIFO order. If the overlay umount fails, every subsequent layer umount returns EBUSY because the overlay still pins them — the previous loop collected and errors.Join()'d every one of those, giving the user a long block of identical "device busy" noise where only the first error described the real problem. Return on the first failed umount with a single error that names which mountpoint the user needs to clear; leave the remaining mounts and the state file in place so a follow-up `apko erofs umount` finishes the job. Deliberately do not fall back to `umount -l`: lazy unmount would let the process exit with the user believing things were torn down while the mounts and pinned files quietly persist. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
erofsmount.Options.ReadOnly was already plumbed into the overlay assembly but had no way to be set from the CLI; the only consumers were external library callers. Wire it through 'apko erofs mount --read-only', omitting the upperdir/workdir overlay just like a library caller would. For single-layer images, overlayfs adds nothing in the read-only case and a lowerdir-only overlay over one EROFS mount has historically been finicky across overlayfs releases. When --read-only is set and the image has exactly one layer, skip the layers/upper/work directories entirely and mount the lone layer straight at DEST/merged. The state file's Mounts slice records that single mountpoint, so Unmount naturally cleans up the same way as a multi-layer mount. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
emitErofsEntry passed mode.Perm() to Mkdir/Mknod/Chmod, and Go keeps setuid/setgid/sticky outside the low 9 bits Perm() returns (fs.ModeSetuid et al are separate high bits), so all three were silently dropped. An EROFS layer built from any normal rootfs shipped a non-setuid sudo, passwd and su, and a non-sticky /tmp. The tar path is unaffected because archive/tar's FileInfoHeader does the translation itself. go-erofs's Writer.Chmod already converts a full fs.FileMode to POSIX mode bits and preserves the entry's type bits, so hand it the unmodified mode once, after the entry exists, instead of at each creation site — Mkdir, Mknod and Create only ever take permission bits. Symlinks are skipped: EROFS pins them at 0777. Verified against erofs-utils: fsck.erofs --extract of an image with a 04755 file, a 02755 file and a 01777 dir yields 4755/2755/sticky after this change and 755/755/755 before it. Note that dump.erofs's "Access:" line masks to 0777 and never shows these bits, even for images built by mkfs.erofs itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Relevant to apko: the reader now reports char devices as fs.ModeDevice|fs.ModeCharDevice, matching Go's own convention (os.Lstat sets both), where 0.3.0 set only ModeCharDevice; Writer errors are now sticky, so a failure inside a long CopyFrom/Create sequence surfaces instead of being dropped; and maxBlockSize is capped at 64 KiB, which images built by apko never exceed (the default 4096 is used). Every apko consumer tests device bits with a mask rather than comparing whole mode values, so the ModeCharDevice change is a no-op here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three display bugs in `apko erofs ls`, all in how the listing reads metadata off the entry: - Every line printed 0/0 for ownership. uidGidFromSys looked for UID()/GID() accessor methods on info.Sys(), but go-erofs returns an *erofs.Stat, which carries them as plain fields — the assertions never matched. This was actively misleading: it read as apko losing ownership when the image was fine. - Devices printed their (zero) inode size where `tar tv` puts major,minor. Decode Stat.Rdev the way Linux's new_encode_dev() wrote it rather than with unix.Major/Minor, whose encoding is host-specific. - setuid/setgid/sticky never rendered. fs.FileInfo.Mode() from the reader carries raw on-disk bits and no Go special-mode bits; Stat.Mode is the translated value. Take mode from there and render the s/S/t/T overloads of the execute columns as `ls -l` does. Entries with no *erofs.Stat — the directories Stack synthesizes for parents no layer contains — keep falling back to the plain FileInfo. Also drop the `ls --help` text describing a temporary mount that has not happened since ls became mount-less, and say plainly that --mode is accepted only for symmetry with `mount` and ignored. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The xattr path in emitErofsEntry was never exercised for anything but user.* on a small fixture, and the package set it was tried against carries no capabilities at all. melange's guest init untars its rootfs with --xattrs-include='security.capability', so an EROFS rootfs that dropped them would break setcap'd binaries with no visible error. Verified working as-is — this test locks it in. It writes a real VFS_CAP_REVISION_2 payload plus a trusted.* and a user.* attribute (three different EROFS name-index prefixes, and one binary value) and requires them back byte-for-byte, then requires the same set from the tar writer over the same source tree so the two layer formats cannot drift. erofs-utils independently agrees the encoding is right: fsck.erofs --extract --xattrs finds security.capability on the inode and declines only because applying it needs root. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DirFS mirrors the backing tree into an in-memory overrides FS at
construction, and those overrides are what every type and mode lookup on
a dirFS resolves against — Lstat reads them directly, Stat takes Mode()
from them, and dirEntry.Type() (what fs.WalkDir hands the tar and EROFS
layer writers) reports them.
The seeding walk dispatched on `switch mode.Type()` with a
`case fs.ModeCharDevice`. Go reports a character device as
ModeDevice|ModeCharDevice — os.Lstat("/dev/null").Mode().Type() has both
bits — so that case never matched and every device node in a pre-existing
tree fell through to the default branch and was seeded as an empty
regular file. A layer built from such a tree ships /dev/null as a
zero-byte file, and Readnod on it fails.
Dispatch on the bit instead. The switch body moves to seedOverride so
the branch is reachable from a test without CAP_MKNOD: the FileInfo comes
from the host's own /dev/null, which is exactly the shape the walk sees.
Two things left as they were, both marked in the code: block devices,
FIFOs and sockets still seed as regular files because the memFS overrides
cannot represent them (apk only ever creates character devices), and
memFS.getNode resolves the final path component, so dirFS.Lstat on a
symlink still reports its target.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An EROFS directory inode's size is the byte length of its dirent blocks, so `ls` printed a number that varied with the child count of whichever single layer won the lookup — for a stack, never the merged directory actually being listed. A two-layer image whose lower layer holds five files in /usr/bin and whose upper holds one printed 41, describing neither. Meanwhile the directories Stack synthesizes for parents no layer contains have no inode at all and reported 0, so one listing mixed both conventions. Print 0 for every directory, which is what the 'tar tvf'-style format this claims to follow does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
formatEntry reads mode, ownership and rdev off the *erofs.Stat behind Sys(). If a future go-erofs returned a different type there, formatEntry would quietly fall back to the plain fs.FileInfo and regress to 0/0 ownership with no setuid/setgid/sticky — the exact pair of bugs fixed in ed0b6c4, reintroduced by a dependency bump with nothing failing. Assert the type, and assert that the UID()/GID() accessors go-erofs documents on the fs.FileInfo agree with the Stat fields the listing actually reads. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review noted the asymmetry: the mkfs.erofs path closes outfile with an error check, while the go-erofs path used a deferred Close plus a Sync, with no stated reason for the Sync. Both paths hash the finished file, so the close has to happen before the hash and its error has to be reported -- a deferred Close would swallow a write error surfaced at close time. Sync was never load-bearing: *os.File does no userspace buffering, so once Close returns a fresh Open sees every byte. Drop it and close explicitly in both branches, with a comment covering the whole block. Also note at both call sites that the discarded second return of CompressionLevel reports whether "level=" was given, not a parse failure; 0 means "let mkfs.erofs choose".
Two silently-dropped errors flagged in review. emitErofsEntry ignored the error from ListXattrs, so a failed xattr lookup produced a file with no xattrs rather than a build failure. apko's FullFS implementations keep xattrs in memory for every node they know about, so an error there means the entry we just walked has gone missing -- a bug worth surfacing. That makes the dirFS path load-bearing, and nothing covered it: every existing erofs test writes from a memFS, while real builds write from a dirFS whose xattr lookups resolve against overrides seeded by walking the backing tree. Add TestWriteErofs_DirFS to pin that down. writeErofsRegularBytes checked Write's error but not its count. The io.Writer contract requires a short write to report an error, but go-erofs's writer is young enough that trusting that silently is not worth it; compare against len(data) and fail with io.ErrShortWrite.
Review asked whether fsckBin is optional, and flagged a discarded error from the lookup helper. It is optional, but nothing said so: three tests ran fsck.erofs only "if err == nil" and passed quietly when erofs-utils was absent. Replace the ad-hoc lookups and lookFsckErofs with optionalFsckErofs, which returns "" and logs when the binary is missing. The doc comment states why it is a second opinion rather than the only check -- every caller has already parsed the image with go-erofs -- and points at TestWriteErofs_FsckErofs as the test that skips outright instead. Also quote the interpolated paths in the roundtrip test's failure messages, per review, so they are easier to pick out of output.
Review asked how the erofs: prefix differs from oci: pointed at an image
with erofs layers, and what happens when erofs: names something that is
not an erofs blob. Document it where the prefixes are defined: the
prefixes select how the path is read, not what the bytes turn out to be,
so erofs: on an OCI layout directory fails at parse time ("not a regular
file"), while erofs: on a non-erofs regular file parses and only fails
at mount time.
Review also asked for specifics behind the claim that a lowerdir-only
overlay over a single erofs mount "has been flaky across overlayfs
versions". We do not have anything concrete to point at, so drop that
sentence and keep only the reason that stands on its own: overlay buys
nothing with one lower and no upper.
smoser
force-pushed
the
feat/apko-erofs
branch
from
August 14, 2026 04:53
1ee8a5b to
2628747
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implement apko writing of erofs images according to draft spec at https://github.com/erofs/erofs-image-spec
The docs/erofs.md file in the pr explains the use and erofs image layout.
I've opened a draft PR showing the value of erofs in images.
See chainguard-dev/melange#2605 for example use of erofs support in apko by melange.
Related: