Conversation
Bluefin Server composes userspace from FSDK 26.08 while taking its kernel and ZFS from Flatcar 4593.2.5. Every installer bring-up defect came from that seam: dracut built the target initrd against a foreign, nested module tree, storage drivers had to be force-loaded by hand, and os-release-flatcar.bst claims ID=flatcar purely so Flatcar sysexts attach. Record the design that collapses the OS payload to a single ABI domain by importing Flatcar's /usr as a pinned binary, keeping BuildStream as the only build system and leaving the installer on FSDK, since systemd-sysinstall is a systemd 261 tool with no Flatcar equivalent. Assisted-by: GPT-5.6 Sol via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
flatcar_production_pxe_image.cpio.gz is not a driver initrd. Measured: four cpio entries wrapping a 374 MiB usr.squashfs, i.e. the whole OS for a RAM boot. Flatcar's real initramfs is /usr/lib/flatcar/bootengine.img, a 50 MiB squashfs with /init and /etc/initrd-release built by sys-kernel/bootengine 0.0.38-r40, shipped inside /usr and versioned with the release. Record the Ignition constraint that comes with it: bootengine drives ignition-fetch/disks/mount/files and sysroot-boot, which this design rejects, so those units must be masked or the fallback is mkosi-initrd --generic. Assisted-by: GPT-5.6 Sol via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
flatcar_production_image_kernel_config.txt declares CONFIG_INITRAMFS_SOURCE="bootengine.cpio" with XZ compression, so the kernel flatcar-kernel.bst already imports is self-sufficient. Flatcar publishes a manifest per stage: 339 entries for the busybox/veritysetup shim that sets up dm-verity for /usr, 2,280 for the systemd initrd it pivots into, matching bootengine.img exactly. flatcar_production_image.vmlinuz and flatcar_production_pxe.vmlinuz are the same 34,245,760 bytes. The installer has therefore been generating a second initrd to lay over a working one. Deleting the dracut step needs no replacement artifact, only conformance to the cmdline that initramfs expects: root=LABEL=ROOT with usr=PARTLABEL=USR-A. That promotes dm-verity A/B from deferred follow-on to the leading option, since it is the only layout upstream tests. Assisted-by: GPT-5.6 Sol via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Booting the pinned flatcar_production_image.vmlinuz in QEMU with no -initrd and no disk reaches "Run /init as init process" at 1.05s, then blocks on "Waiting for drive..." while stage 1 hunts for usr=PARTLABEL=USR-A. That demonstrates the initramfs is compiled in and live rather than merely declared, and shows the boot contract asserting itself. The same trace also shows the kernel prepends a built-in CONFIG_CMDLINE of "rootflags=rw mount.usrflags=ro", so a UKI cmdline merges with it rather than replacing it. Assisted-by: GPT-5.6 Sol via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Decision: follow Flatcar's design. The OS payload becomes a /usr image on a verity-protected A/B pair and / becomes writable state. Overriding the built-in initramfs, by dracut or mkosi-initrd, is rejected. Extracted stage 1's /init from the kernel's embedded initramfs and read the contract instead of guessing it. veritysetup runs with a hardcoded --hash-offset=1065345024, so verity is one partition with the hash tree appended, not systemd-repart's two-partition model; the image carries its own hash and repart keeps using CopyBlocks. Verity is skipped when verity.usrhash is empty, so layout and verity split into separate tickets. mount -t passes usrfstype through, so /usr stays XFS. Stage 1 loop-mounts /usr/lib/flatcar/bootengine.img from the mounted /usr, so stage 2 ships with the payload and must survive the import strip list. Records Flatcar's GPT read from flatcar_production_image.bin and maps it onto repart.d: EFI-SYSTEM, USR-A, USR-B, ROOT, with BIOS-BOOT, OEM, and OEM-CONFIG dropped as BIOS-only or Ignition-only. Assisted-by: GPT-5.6 Sol via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Built a conforming disk from upstream artifacts alone, unprivileged: GPT with Flatcar's type GUIDs, /usr as a 356 MiB squashfs in USR-A, ext4 OEM and ROOT via mke2fs -d. The pinned kernel booted it with no external initrd through stage 1, bootengine, switch_root, and on to multi-user.target and a login prompt, with SSH host keys and DHCP up. Two corrections from the trace. OEM is required, not droppable: stage 2 waits on dev-disk-by-label-OEM.device, matched by filesystem label rather than partition label, and drops to an emergency shell after 90s without it. Ignition needs no masking: ignition-setup-pre completes, delete-config skips on its trigger condition, and ignition-subsequent.target reports "Not Ignition" boot complete. Also records the competing version-parity plan. Adopts its pin-discipline, systemd-creds provisioning, and reboot coordination. Rejects rebuilding Flatcar versions from source, since version parity is not ABI parity under a different toolchain; rejects three tarball-built kernels, which would each lose CONFIG_INITRAMFS_SOURCE and re-create the initrd problem; rejects dropping k0s, which the KubeStellar path depends on; and supersedes its root-a/root-b model with the measured USR-A/USR-B pair. Assisted-by: GPT-5.6 Sol via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The objective is version parity, not ABI parity, and the question the project answers is what Flatcar Container Linux looks like built with BuildStream. For each component Flatcar ships, build that same upstream version from source. Binaries will differ because the toolchain differs - Flatcar's kernel banner reads Gentoo Hardened gcc 14.3.1 - and that is accepted. Success is that the image boots and behaves and that every divergence is recorded, not that bytes match. The build base stays freedesktop-sdk 26.08 and hard rule 1 is unchanged. Flatcar's binaries are imported as a reference to boot against and measure, not as the shipped payload, so the method is import first then substitute, with the boot proof as the control. Release version becomes 26.08.XX.$FLATCARVERSION, one string with both halves enforced: the FSDK point release against the junction ref, the Flatcar release against the pin in include/flatcar.yml. This supersedes the earlier two-field proposal. Adds a version audit phase producing a parity matrix of component, Flatcar version, FSDK version, gap, which orders all downstream work. Assisted-by: GPT-5.6 Sol via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
castrojo
changed the title
release-version becomes the Flatcar release the image targets, 4593.2.5, not a composite and not a translation. If the image claims parity with Flatcar 4593.2.5 it says so. The tree is already half-way there and inconsistent with itself: os-release-flatcar.bst sets VERSION_ID and CPE_NAME from FLATCAR_VERSION while PRETTY_NAME carries the unrelated release-version, so one image answers two different questions about its own version. This collapses them. It also makes sysext attachment honest. Flatcar sysexts match on ID and VERSION_ID, and flatcar-zfs.raw attaches today only because the identity file already asserts Flatcar's. With release-version equal to the Flatcar version the match rests on a true statement. check-release-version.py inverts: it stops enforcing against the FSDK junction ref and starts enforcing against flatcar-version in include/flatcar.yml. The FSDK pin stays for provenance in fsdk_ref and gets its own os-release field rather than being smuggled into the version string. Prerelease suffixes keep working, so an alpha is 4593.2.5-alpha.1. Supersedes both the two-field split and the 26.08.XX.$FLATCARVERSION composite. Assisted-by: GPT-5.6 Sol via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
hanthor
left a comment
hanthor
left a comment
There was a problem hiding this comment.
This document contradicts the four implementation PRs open against it, and in places contradicts itself. As written it can't serve as the design of record for #132/#133/#136/#140/#141.
1. Hard rule 1: stated as unchanged, three times
The build base stays freedesktop-sdk 26.08. Hard rule 1 is unchanged: compose from FSDK
components/*. Version parity is pursued within the FSDK graph, by building the versions Flatcar ships rather than by importing Flatcar's tree wholesale.
Both images are composed from FSDK 26.08. […] Installed OS payload tracks Flatcar's versions […] still built from FSDK
components/*.
Hard rule 1 is untouched throughout: the base stays FSDK 26.08.
#133 changes that rule to "Installer image composes from FSDK 26.08 components/*; OS payload may import Flatcar binaries pinned by digest," and its ADR explicitly rejects the hybrid this document describes: "Keeping FSDK userspace in the OS payload while importing Flatcar kernel and ZFS artifacts preserves the ABI split that the migration is meant to remove."
2. Internally inconsistent
The paragraph above says "rather than by importing Flatcar's tree wholesale." Two sections later, ### New element: flatcar/flatcar-usr.bst specifies exactly that import, and ### Overlay: what Flatcar displaces tabulates the removal of the entire FSDK base runtime. Phase 3 then says the imports are "the oracle, not the payload" — but #132's os-stack.bst makes flatcar-usr.bst the payload.
These are two different designs in one file, and the reader can't tell which one governs.
3. ABI parity
This says "ABI parity is explicitly not the goal […] the binaries will differ, and that is expected and accepted." #132's element header says it "Replaces the FSDK 26.08 base userspace […] with a single-ABI Flatcar /usr." Opposite objectives.
4. §Versioning explicitly supersedes #136
.github/scripts/check-release-version.pyinverts. It stops enforcingrelease-versionagainst the FSDK junction ref and starts enforcing it againstflatcar-version.This supersedes both earlier proposals: the two-field split and the
26.08.XX.$FLATCARVERSIONcomposite.
#136 is the two-field split (installer-version + flatcar-version) and it keeps enforcing installer-version against the junction ref. One of the two has to be withdrawn.
For what it's worth I think #136 has the better of the argument — this document's own "the installer and the payload diverge in version" section is an argument for two axes, since systemd-sysinstall only exists in the installer's FSDK systemd and the installer genuinely cannot follow the payload's version. But that reasoning supports two axes, not collapsing onto one.
Two things here the implementations should adopt, and one that's factually wrong
Adopt: "Removals are explicit rm lines with a comment naming the replacement, not a wildcard sweep, so a future Flatcar bump that renames a unit fails loudly." #140 follows this exactly. #132's single rm -rf block does not — and I confirmed against the real tarball that three of its paths don't exist and fail silently, while six real ones go unremoved.
Adopt: "Contract tests assert /usr/lib/flatcar/bootengine.img is present and the module layout is flat." #140 has these; #132 doesn't.
Wrong: the design (and both implementations) assume Flatcar nests modules at usr/lib/modules/<kver>/<kver>/. I listed all 19,658 entries of flatcar-container.tar.gz (sha256 c533ff57…, the ref both PRs pin) — the tree is flat. There is no nested directory to collapse, so the flattening block is dead code in both. The likely origin of the belief is the --strip-components=5 in the existing flatcar-kernel.bst.
Also missing
The "what Flatcar displaces" table maps podman → flatcar-podman.raw sysext, but #141 additionally imports containerd and docker, which this document doesn't mention at all. And the document says the target dracut invocation "is deleted, and nothing replaces it" — none of #132/#140/#141 touches elements/oci/bluefin-server-installer.bst, where that invocation lives.
docs-checks.py passes and pytest/bats are unaffected — this is a documentation-consistency objection, not a test failure.
Generated by Claude Code
|
Thanks for the thorough review. The core tensions you identified are valid and reflect the ongoing discussion around the migration strategy:
|
2 participants
Summary
Design for version parity with Flatcar Container Linux: for each component Flatcar ships, build that same upstream version from source in BuildStream, on the freedesktop-sdk 26.08 base.
The question the project answers: what would Flatcar look like if it were built with BuildStream?
Version parity, not ABI parity
x86_64-cros-linux-gnu-gcc (Gentoo Hardened 14.3.1_p20250801 p4). Our binaries will differ. Accepted.Done is not "our bytes match theirs". Done is "we build the versions Flatcar builds, the result boots and behaves, and we can say precisely where a BuildStream-built Flatcar diverges from the real one."
Hard rule 1 is unchanged. The base stays FSDK 26.08. Flatcar binaries are imported as a reference to boot against and measure, never as the shipped payload.
Versioning
Both halves enforced: FSDK point release against the junction ref, Flatcar release against the pin in
include/flatcar.yml.Method: import first, then substitute
Parity needs an oracle. Import Flatcar's binaries, boot them on our installer and layout to establish a known-good control, then replace components one at a time with FSDK-built ones at Flatcar's versions, re-running the boot proof after each swap.
Boot proof (already achieved)
Built a conforming disk from upstream artifacts alone, unprivileged —
sfdisk,mksquashfs,mke2fs -d, no root, no loop mounts — and booted the pinned kernel with no external initrd:Findings that drove the design
CONFIG_INITRAMFS_SOURCE="bootengine.cpio"— the kernel already carries a two-stage initramfs. The installer has been generating a second one to lay over a working one.veritysetup --hash-offset=1065345024is hardcoded in stage 1: verity is one partition with the hash appended, not systemd-repart's two-partition model./usrhas a hard 1,065,345,024-byte budget.bootengine.img, loop-mounted from the mounted/usr— so it versions with the payload, and must survive the import.OEMis required, matched by filesystem label; without it the boot stalls 90s then drops to emergency.Subsequent (Not Ignition) boot complete.USR-A/USR-B(5dfbf5f4-2848-4bac-aa5e-0d9a20b745a6), notroot-a/root-b.Scope
Design only, no element changes. Ticket chain: #125 versioning · #137 pins · #142 parity matrix · #126 reference import · #134 layout · #135 verity · #128 dracut removal · #129 comparison harness · #130 boot proof in CI · #138 creds · #139 reboot coordination.
Assisted-by: GPT-5.6 Sol via GitHub Copilot